Custom Imagery Scripts for GOES
“Extinction is the rule. Survival is the exception.” – Carl Sagan
- ABI Information
- Creating an Animation with ImageMagick and FFMPEG
- Composite Imagery using layers
- A new way to composite using transparent layering
- Sanchez- using a false-color static image to ‘colorize’ a greyscale IR Image
- Reprojecting a global image to a flat equirectangular image with “Sanchez”
- Sanchez in Virtual Satellite mode
- Using Xplanet to create a day/night underlay.
- Pushing the Limit- or trying to..
- Rayleigh and Specular Lighting effects
- GOES Full-Disc Animations in Linux
UPDATED 02/26/2023
I am posting the scripting I use for compositing, cropping, and animating GOES and other… received imagery using ImageMagick and FFMPEG and Sanchez, Powershell, WinSCP, JREPL .
Recently I took 6 months 1 day (185 days) of imagery. 8,780 worth of received GOES 16 Images, and created this animation.
First (FAST) version upscaled to 42234 frames run time 5:14
Second (SLOWER) version upscaled to 62482 frames run time 31:21
GOES provides a lot of imagery!
The ABI can multitask. The default scan mode concurrently takes a full disk image every 10 minutes, an image of the contiguous U.S. every 5 minutes, and two smaller, more detailed images of areas where storm activity is present, every 60 seconds (or one every 30
seconds). Alternatively, ABI can run in full-disk mode, continuously imaging the full disk every 5 minutes.
Now, with the ABI imagery capabilities listed, why is that on HRIT, we do not get those cool 10-minute full disk scans, but only get them every thirty minutes? The answer is bandwidth. There is a limited amount of data that can be sent, so we, who receive the HRIT and not GRB get a bit downgraded timing and imagery.
Creating an Animation
So we get all these cool images, but how to stitch them together into some semblance of animation? The first, and easiest way is through GIF animation. Take a bunch of timed pictures, assemble them together frame by frame and use a gif animator to make a “moving’ image. There are many online gif animation sites that are quite good like https://gifmaker.org/ or https://gifmaker.me/ But, because the images we get are 15 to 30 minutes apart. they can appear jerky or stuttered. (Imagine watching Star Wars at 1 frame every 30 minutes instead of one frame every 24 seconds!).
So here is an example of a full disk compiled into a gif animation, nice, but still kind of jerky. After all, each “snapshot” of the planet is taken every 30 minutes.
So how do we try and smooth this out? This is where magic comes in, in the form of ImageMagick and FFMPEG software. Below is an example of a gif animation turned into a smooth-flowing mp4 movie using a combination of Imagemagick and FFMPEG Scripting. The first script compiles all the frames, and then layers channel 15 over the full-color earth. The second script, using FFMEG, then takes the compiled gif animation and using a video filter called minterpolate, which performs motion interpolation. It’s a tool that can: Increase a video’s frame rate by rendering new frames based on motion (pixel-warping + blending). Make smooth motion videos (simple blending between frames) between the frames So basically it is filling in the 30-minute gap in full-disk imagery. Rember all this scripting is based on the way my system is set up, with multiple drives, etc. Your script will, must, be changed to match your own system!
So, what’s the secret sauce that creates this magic?
I Use ImageMagick and FFMPEG on my Windows 10 Machines. What this does is allows me to write Batch Files (example.BAT) that I can run by double-clicking the chosen script in my “MagickScripts” Folder.
Since I do a lot of different types of animation, I divided my folder structure into many types of animations, such as F:/Fulldiskfile, F:/Conusfile, F:/CMIPFILE , etc. Under each one of these, I have subfolders where the program stores the temporary files used in creating the animations. As an example, one full disk animation with one or two overlays of ABI Band imagery might contain 4,500 separate images depending on the length of the capture!
Full Disk Animation
So here is one example of a ‘simple’ Full Disk animation, that does the following:
(Remember I run my scripts as Windows Batch Files, Your application and folder structure may (will) be different!!).
- Reduces the file size while maintaining the best balance of color and sharpness.
- Reduces the image size from an as-received image of 5424×5424 (that’s 75 inches wide and tall!) to 2000×2000 pixels
- It gives me the option to manually delete night time images (I did this for the animations that DO NOT have any layering of other bands).
- Adds my watermark (“branding”, you would have to create your own transparent png) to my image.
- Creates a GIF of all the reformatted images
- Then creates an Mp4 with interpolation of the frames.
- Then it deletes the temp files.
REM FULL DISK ANIMATION REM STEP #1 AND #2 - REDUCE FILESIZE for /r "F:\Satellite Imagery\goes16\fd\NOMAP\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path "F:\FULLDISKFILE\FULLDISK\AUTORESIZE" -filter Triangle -define filter:support=2 -thumbnail 2000 -unsharp 0.25x0.25+8+0.065 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB -strip "%%~a" REM STEP #3 OPTION TO DELETE NIGHT TIME IMAGERY ECHO Manually delete night imagery in /FULLDISKFILE/FULLDISK/AUTORESIZE/? PAUSE REM STEP #4 ADD WATERMARK TO REDUCED FULL DISK IMAGE magick mogrify -path "F:/FULLDISKFILE/FULLDISK/FDWATERMARKED/" -format jpg -gravity northwest -draw "image over 30,30 0,0 'F:/Satellite Imagery/GOESFULLDISKWHITE500x100.png'" "F:/FULLDISKFILE/FULLDISK/AUTORESIZE/*.jpg" REM STEP #5 CREATE GIF magick convert -loop 0 -delay 20 "F:\FULLDISKFILE\FULLDISK\FDWATERMARKED\GOES16_*.jpg" "F:/FULLDISKFILE/FULLDISK/FULLDISKFINAL/NOMAPout.gif" REM STEP #6 CREATE MP4 ffmpeg -i "F:/FULLDISKFILE/FULLDISK/FULLDISKFINAL/NOMAPout.gif" -filter "minterpolate='mi_mode=blend'" -c:v libx264 -pix_fmt yuv420p "F:/FULLDISKFILE/FULLDISK/FULLDISKFINAL/NOMAPGOES.mp4" REM STEP #7 DELETE TEMPORARY FILES del "F:\FULLDISKFILE\FULLDISK\AUTORESIZE\*.jpg" /s /f /q del "F:\FULLDISKFILE\FULLDISK\FDWATERMARKED\*.jpg" /s /f /q echo Your file is finished
NOTE: My file structure will obviously be different than your own! And also note that because this bat file is calling out ImageMagick and FFMPEG to run, you will see some commands in quotes “” and also note the difference in file structuring with the “/” vs. the “\” signs. They make a huge difference and need to be correct! Also, take note that in steps 1 and 2 above I am using a Recursive For Loop to loop through the directories where necessary to modify every image inside every dated folder.
Imagine the for loop is being put “on pause” while the function (in this case reducing file size and reducing pixel size) call is running .
The new recursive call starts its for
loop and again pauses while calling the functions again, and so on until every subdirectory and file has been modified.
Once you have the basics down, lines of this code can be used in multiple scripts, I have 43 batch files and counting! One for CONUS, one for Compositing, one for fulldisk, one for Meso, etc.
This bit of code I use in many of my scripts is the one that does the file size reduction and pixel size reduction. My File size reducer and optimizer script (Note the OUTPUT_PATH, OUTPUT_WIDTH, and INPUT_PATH) :
REM FILE SIZE REDUCTION mogrify -path OUTPUT_PATH -filter Triangle -define filter:support=2 -thumbnail OUTPUT_WIDTH -unsharp 0.25x0.25+8+0.065 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB -strip INPUT_PATH
Here are the two other lines of code that take the images and first create the GIF, then the MP4 which is “smoothed out” through the minterpolate function of the script:
::CREATE GIF magick convert -loop 0 -delay 20 "C:/INPUT_FILE_DIRECTORY/*.jpg" "C:/OUTPUT_FILE_DIRECTORY/FILENAME_OF_GIF.gif" ::CREATE MP4 ffmpeg -i "C:/OUTPUT_FILE_DIRECTORY/FILENAME_OF_GIF.gif" -filter "minterpolate='mi_mode=blend'" -c:v libx264 -pix_fmt yuv420p "C:/OUTPUT_FILE_DIRECTORY/FILENAME_OF_MP4.mp4"
Compositing Full-Color Imagery with Band 15 Enhanced Imagery at 120FPS
Here is the latest result in my attempt to smooth out and interpolate the final video animations I create from GOES 16 Imagery. GOES 16 and 17 transmit HRIT imagery full disk every half hour or 2 frames every 3600 seconds. (2FPH (Frames Per Hour))
Compare that to a standard video that runs at 24 frames every second (24 FPS or 86400FPH).
The easiest method to stream the individual frames is to decrease the time in a GIF animation to about a 20th of a second between frames, rendering a shorter time span for the compiled gif and making the resulting gif “smoother but still jumpy”, after all, there is a half-hour gap in realtime between frames.
So, how to improve the transition between the frames?
Depending on the image source (M1, M2, Full Disks, what Band, etc.) I use a combination of three tools
Irfanview – to Date stamp individual frames if necessary
Image Magick – to crop, resize, rename, watermark, compress, set color space and create the gif animation, compose, blend, and composite
FFmpeg – To create the final mp 4 video through the use of the motion interpolation modes.
It is through the interpolation mode in FFmpeg, where most of the ‘Magick” happens. I tried using “Butterflow” but the results were not as smooth and it took hours to run on my computer. So I stuck with FFmpeg.
The FFmpeg script I wrote first takes the compiled gif animation and:
- Duplicates the previous and next frame for interpolating new frames.
- Adds Bidirectional motion estimation. Motion vectors are estimated for each source frame in both forward and backward directions.
- Adding Motion compensation mode. Using overlapped block motion compensation.
1, 2, and 3 together create “new” filler frames based on the information in the original frames thereby filling the 30-minute gaps between frames with estimated imagery based on every previous and nest frame. (Whew!) - The algorithm I chose which seems to work best for this type of motion estimation is the Diamond search algorithm.
- The video is also increased from the default of 60 FPS to 120 FPS and then rendered into a format that most video players can handle.
All of the above steps are performed in a single line of the script. And it is a balance between the time it takes to create the animation and the resulting file size.
Batch File for Creating a video of FC with my custom CLUT and Channel 15 Enhanced and a choice for creating a CONUS animation as well:
REM USRADIOGUY Build Imagery for Compositing FC and 15Enhanced 8/1/2020
REM MOVE AND RESIZE AND RENAME
for /r "F:\Satellite Imagery\goes16\fd\CUSTOMLUT\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../COMPOSITEFILE/compositeFC -filter Triangle -define filter:support=2 -thumbnail 1000 -unsharp 0.25x0.25+8+0.065 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB -strip -gravity south -stroke none -pointsize 24 -fill white -font Ailerons -annotate +0+0 GOES_16_FC_and_Channel_15 "%%~a"
for /r "F:\Satellite Imagery\goes16\fd\ch15_enhanced\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../COMPOSITEFILE/composite15 -filter Triangle -define filter:support=2 -thumbnail 1000 -unsharp 0.25x0.25+8+0.065 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB -strip -gravity south -stroke none -pointsize 24 -fill white -font Ailerons -annotate +0+0 GOES_16_FC_and_Channel_15 "%%~a"
@echo off
echo [101;93m CHECK FILE COUNT TO BE SURE IT"S EQUAL BEFORE CONTINUING IN FOLDER F:\COMPOSITEFILE\compositeFC\ AND F:\COMPOSITEFILE\composite15\ AND ADJUST IF NEEDED [0m
Pause
echo START COMPOSITING False Color into Channel_15 Enhanced
setlocal enabledelayedexpansion
set dirA=F:\COMPOSITEFILE\compositeFC\
set dirB=F:\COMPOSITEFILE\composite15\
set dirOut=F:\COMPOSITEFILE\compositedimages\
dir /b /on %dirA%*.jpg >listA.lis
set cntA=0
for /F "tokens=*" %%F in (listA.lis) do (
set listA[!cntA!]=%%F
set /A cntA+=1
)
echo There are %cntA% file in list A.
dir /b /on %dirB%*.jpg >listB.lis
set cntB=0
for /F "tokens=*" %%F in (listB.lis) do (
set listB[!cntB!]=%%F
set /A cntB+=1
)
echo There are %cntB% file in list B.
if not %cntA%==%cntB% (
echo The lists have different numbers of files.
exit /B 1
)
set /A lastN=%cntA%-1
for /L %%N in (0,1,%lastN%) do (
echo %%N: composite %dirB%!listB[%%N]! and %dirA%!listA[%%N]!
rem magick.exe composite -blend 50 %dirB%!listB[%%N]! %dirA%!listA[%%N]! %dirOut%!listA[%%N]!
magick %dirA%!listA[%%N]! %dirB%!listB[%%N]! -define compose:args=55 -compose blend -composite %dirOut%!listA[%%N]!
rem magick %dirA%!listA[%%N]! { %dirB%!listB[%%N]! -transparent black } -define compose:args=50 -compose blend -composite %dirOut%!listA[%%N]!
)
REM ADD WATERMARK TO THE REDUCED FULL DISK IMAGE
magick mogrify -path "F:/COMPOSITEFILE/compositefinal/" -format jpg -gravity northwest -draw "image over 15,15 0,0 'F:/Satellite Imagery/GOESFULLDISKWHITE250x50.png'" "F:/COMPOSITEFILE/compositedimages/*.jpg"
REM CREATE GIF
magick convert -loop 0 -delay 25 "F:\COMPOSITEFILE\compositefinal\GOES16_*" "F:/COMPOSITEFILE/compositefinal/compositeout.gif"
REM CREATE MP4
ffmpeg -i "F:/COMPOSITEFILE/compositefinal/compositeout.gif" -filter "minterpolate='mi_mode=blend:me_mode=bidir:fps=120:mc_mode=obmc:me=ds:vsbmc=1'" -c:v libx264 -pix_fmt yuv420p "F:/COMPOSITEFILE/compositefinal/compositeout120fps.mp4"
@ECHO OFF
:start
SET choice=
SET /p choice=Do you want to make a CONUS animation? [N]:
IF NOT '%choice%'=='' SET choice=%choice:~0,1%
IF '%choice%'=='Y' GOTO yes
IF '%choice%'=='y' GOTO yes
IF '%choice%'=='N' GOTO no
IF '%choice%'=='n' GOTO no
IF '%choice%'=='' GOTO no
ECHO "%choice%" is not valid
ECHO.
GOTO start
:no
ECHO I guess no one gets to see the cool stuff!
del "F:\CONUSFILE\CONUSCROPPED\*.*" /s /f /q
del "F:\COMPOSITEFILE\composite15\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositeFC\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositedimages\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositefinal\*.jpg" /s /f /q
PAUSE
EXIT
:yes
ECHO Creating CONUS now!
REM After using CompositingFCandch15.bat
REM CROP ALL IMAGES IN F:\Satellite Imagery\goes16\fd\NOMAP\
REM FOR 1000X1000 only!!!
for /r "F:\COMPOSITEFILE\compositefinal\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../CONUSFILE/CONUSCROPPED -format jpg -crop 618x376+21+5 "%%~a"
REM FOR 2000X2000 only!!!
REM for /r "F:\COMPOSITEFILE\compositefinal\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../CONUSFILE/CONUSCROPPED -format jpg -crop 1133x625+235+55 "%%~a"
REM CREATE GIF
magick convert -loop 0 -delay 15 "F:\CONUSFILE\CONUSCROPPED\GOES16_*" "F:/CONUSFILE/CONUS/CONUSfinal/COMPOSITECONUSout.gif"
REM CREATE MP4 WITH INTERPOLATIN
ffmpeg -i "F:/CONUSFILE/CONUS/CONUSfinal/COMPOSITECONUSout.gif" -filter "minterpolate='mi_mode=blend:me_mode=bidir:fps=120:mc_mode=obmc:me=ds:vsbmc=1'" -c:v libx264 -pix_fmt yuv420p "F:/CONUSFILE/CONUS/CONUSFinal/COMPOSITECONUS120FPSout.mp4
REM DELETE ALL TEMP FILES
del "F:\CONUSFILE\CONUSCROPPED\*.*" /s /f /q
del "F:\COMPOSITEFILE\composite15\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositeFC\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositedimages\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositefinal\*.jpg" /s /f /q
PAUSE
EXIT
New script for compositing False Color and Band 15 Enhanced 10.15.2020
So, as I have written, this page is a work in progress. The compositing script above, to me, always left something to be desired on the final output. The final animation never quite allowed the daytime image to be a clear and sharp as I wanted. So I investigated ways to completely make the background transparent on the GOES CH15 Enhanced so that when layered, it would not dull down the background Full-Color Disk. I found the way to do this via an Image Magick command ” -transparent” By using this command in my script and converting the imagery from jpg, and outputting it in a png format, I can remove the black background (that I specify in my GOESPROC only for ch15e). This then allows the resulting layering to be quite precise and clean. The drawback is, it takes much longer to run the script, and the resulting mp4 file size is much larger.
Here is the script that I created.
REM USRADIOGUY Build Imagery for Compositing 10/12/2020
set startTime=%time%
REM MOVE AND RESIZE AND RENAME CUSTOMLUT
for /r "F:\Satellite Imagery\goes16\fd\CUSTOMLUT\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../COMPOSITEFILE/compositeFC -type TrueColor -filter Triangle -define filter:support=2 -thumbnail 2000 -unsharp 0.25x0.25+8+0.065 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -interlace none -colorspace sRGB -strip -gravity south -stroke none -pointsize 24 -fill white -font Ailerons -annotate +0+0 GOES_16_FC_and_Channel_15 "%%~a"
REM MOVE AND RESIZE AND RENAME CREATE TRANSPARENCY,AND SAVE AS PNG
for /R "F:\Satellite Imagery\goes16\fd\ch15_enhanced" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../COMPOSITEFILE/composite15 -filter Triangle -define filter:support=2 -thumbnail 2000 -unsharp 0.25x0.25+8+0.065 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB -strip -gravity south -stroke none -pointsize 24 -fill white -font Ailerons -annotate +0+0 GOES_16_FC_and_Channel_15 -format png -resize 2000 -transparent black -fuzz 2%% "%%~a
echo [101;93m CHECK FILE COUNT TO BE SURE IT"S EQUAL BEFORE CONTINUING IN FOLDER F:\COMPOSITEFILE\compositeFC\ AND F:\COMPOSITEFILE\composite15\ AND ADJUST IF NEEDED [0m
Pause
echo Start compositing False Color into Channel_15 Enhanced into temp files
setlocal enabledelayedexpansion
set dirA=F:\COMPOSITEFILE\compositeFC\
set dirB=F:\COMPOSITEFILE\composite15\
set dirOut=F:\COMPOSITEFILE\compositedimages\
dir /b /on %dirA%*.jpg >listA.lis
set cntA=0
for /F "tokens=*" %%F in (listA.lis) do (
set listA[!cntA!]=%%F
set /A cntA+=1
)
echo There are %cntA% file in list A.
dir /b /on %dirB%*.png >listB.lis
set cntB=0
for /F "tokens=*" %%F in (listB.lis) do (
set listB[!cntB!]=%%F
set /A cntB+=1
)
echo There are %cntB% file in list B.
if not %cntA%==%cntB% (
echo The lists have different numbers of files.
exit /B 1
)
set /A lastN=%cntA%-1
for /L %%N in (0,1,%lastN%) do (
echo %%N: composite %dirB%!listB[%%N]! and %dirA%!listA[%%N]!
rem magick.exe composite -blend 50 %dirB%!listB[%%N]! %dirA%!listA[%%N]! %dirOut%!listA[%%N]!
magick %dirA%!listA[%%N]! %dirB%!listB[%%N]! -define compose:args=55 -compose blend -composite %dirOut%!listA[%%N]!
rem magick %dirA%!listA[%%N]! { %dirB%!listB[%%N]! -transparent black } -define compose:args=50 -compose blend -composite %dirOut%!listA[%%N]!
)
REM ADD USRADIOGUY.COM WATERMARK TO THE REDUCED FULL DISK IMAGE
magick mogrify -path "F:/COMPOSITEFILE/compositefinal/" -format jpg -gravity northwest -draw "image over 15,15 0,0 'F:/Satellite Imagery/GOESFULLDISKWHITE250x50.png'" "F:/COMPOSITEFILE/compositedimages/*.jpg"
REM CREATE GIF
magick convert -loop 0 -delay 20 "F:\COMPOSITEFILE\compositefinal\GOES16_*" "F:/COMPOSITEFILE/compositefinal/compositeout.gif"
REM CREATE FILE TIMESTAMP
set year=%date:~10,4%
set month=%date:~4,2%
set day=%date:~7,2%
set curTimestamp=%month%%day%%year%
REM CREATE MP4 with Timestamp
ffmpeg -i "F:/COMPOSITEFILE/compositefinal/compositeout.gif" -filter "minterpolate='mi_mode=blend:me_mode=bidir:fps=120:mc_mode=obmc:me=ds:vsbmc=1'" -c:v libx264 -pix_fmt yuv420p "F:/COMPOSITEFILE/compositefinal/compositeout120fps_%month%%day%%year%.mp4"
echo Start Time: %startTime%
echo Finish Time: %time%
pause
@ECHO OFF
:start
SET choice=
SET /p choice=Do you want to make a CONUS animation? [N]:
IF NOT '%choice%'=='' SET choice=%choice:~0,1%
IF '%choice%'=='Y' GOTO yes
IF '%choice%'=='y' GOTO yes
IF '%choice%'=='N' GOTO no
IF '%choice%'=='n' GOTO no
IF '%choice%'=='' GOTO no
ECHO "%choice%" is not valid
ECHO.
GOTO start
:no
ECHO I guess no one gets to see the cool stuff!
del "F:\CONUSFILE\CONUSCROPPED\*.*" /s /f /q
del "F:\COMPOSITEFILE\composite15\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositeFC\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositedimages\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositefinal\*.jpg" /s /f /q
PAUSE
EXIT
:yes
ECHO Creating CONUS now!
REM CROP Imagery into a CONUS PROJECTION - After using CompositingFCandch15.bat
REM CROP ALL IMAGES IN F:\Satellite Imagery\goes16\fd\NOMAP\
for /r "F:\COMPOSITEFILE\compositefinal\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../CONUSFILE/CONUSCROPPED -format jpg -crop 618x376+21+5 "%%~a"
REM FOR 2000X2000 for /r "F:\COMPOSITEFILE\compositefinal\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../CONUSFILE/CONUSCROPPED -format jpg -crop 1134x624+234+54 "%%~a"
REM CREATE GIF
magick convert -loop 0 -delay 15 "F:\CONUSFILE\CONUSCROPPED\GOES16_*" "F:/CONUSFILE/CONUS/CONUSfinal/COMPOSITECONUSout.gif"
REM CREATE MP4
ffmpeg -i "F:/CONUSFILE/CONUS/CONUSfinal/COMPOSITECONUSout.gif" -filter "minterpolate='mi_mode=blend:me_mode=bidir:fps=120:mc_mode=obmc:me=ds:vsbmc=1'" -c:v libx264 -pix_fmt yuv420p "F:/CONUSFILE/CONUS/CONUSFinal/COMPOSITECONUS120FPSout_%month%%day%%year%.mp4
REM DELETE TEMP FILES
del "F:\CONUSFILE\CONUSCROPPED\*.*" /s /f /q
del "F:\COMPOSITEFILE\composite15\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositeFC\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositedimages\*.*" /s /f /q
del "F:\COMPOSITEFILE\compositefinal\*.jpg" /s /f /q
PAUSE
EXIT
Sanchez brings your dull IR satellite images to life.
Matt Painter has created a slick piece of software called Sanchez which can be downloaded from his GitHub repository. Having heard about it from him on Reddit I knew I had to give it a try for my dull GOES 17 IR Imagery (NOTE: it will work on Himawari, GK2A GOES 16, and GOES 18, Elecktro Ln2, and EWS-G1 and more).
Utilizing a full-color high resolution, static underlay image, and combining it with a single greyscale IR satellite image, such as I get relayed from GOES 18, along with adding an optional mask and overlay and some coding magic, Sanchez will create beautiful images to be proud of.
Sanchez has been updated with:
- Properly projected underlays
- Underlays Global Coverage
- Batch conversion mode with glob support
- Overlay support
- Raspberry Pi support
- Bug fixes
Here is a simple Batch File I wrote to use Sanchez on my windows 11 Machine.
REM CREATE A TIME STAMP
set year=%date:~10,4%
set month=%date:~4,2%
set day=%date:~7,2%
set curTimestamp=%month%%day%%year%
REM RUN SANCHEZ
"F:/Sanchez/Sanchez.exe" -s "F:\YOUR_PATH\TO_IMAGES\GOES18_FD_CH13_20201206T140013Z.jpg" -v -u "F:\Sanchez\Resources\world.200411.3x10848x5424" -o "F:\YOUR_PATH\TO_OUTPUT_IMAGES\GOES16_OUTPUT_%month%%day%%year%_.jpg"
pause
The resulting Image!
Or using Himawari-9 as an example:
REM CREATE A TIME STAMP
set year=%date:~10,4%
set month=%date:~4,2%
set day=%date:~7,2%
set curTimestamp=%month%%day%%year%
REM RUN SANCHEZ
F:/Sanchez/Sanchez.exe -s "F:\YOUR_PATH\TO_IMAGES\Himawari9_FD_IR_20230701T045100Z.jpg" -v -u "F:\Sanchez\Resources\world.200411.3x10848x5424" -o "F:\YOUR_PATH\TO_OUTPUT_IMAGES\Himawari9_FD_IR_OUTPUT_%month%%day%%year%_.jpg"
pause
Sanchez is a software tool that GOES, GK2A, Himawari, Electro, EWS-G2 (actually any geo sat if you configure it) Satellites to be combined, reprojected, projected, stitched onto a globe, either creating a flat projected image or by creating a ‘virtual’ satellite image at a given longitude.
The new version has a ton of possibilities and allows geostationary satellite imagery to be projected in a Plate Carrée projection & allows stitching of multiple sats, GOES 16, GOES 17, GK2A, Himawari, etc, It is primarily for enhancing IR imagery onto static imagery such as ‘blue marble’ or even night views, but it really ups the game for geostationary satellite observation. And the code runs fast. I am already adapting it to reproject some of my sandwich layer imagery.
I have just begun to experiment by combining my scripts with the Sanchez code. Adding animation into the mix, and the results are quite impressive!
So how does one use Sanchez? Well, the author is building a WIKI as he creates the software. You can find it here https://github.com/nullpainter/sanchez/wiki. Releases are available for Raspberry Pi, Linux, Mac OS X and Windows here https://github.com/nullpainter/sanchez.
Sanchez basically works by pointing the software at the directory or images you wish to process and specifying the output file or name. Then, by a system of comand arguments within the script you can select, reprojection, an underlay, or no underlay, timestamps, interpolation, histogram equalization, and more.
A couple of sample scripts using Sanchez:
I would first STRONGLY encourage you to read through the Sanchez WIKI https://github.com/nullpainter/sanchez/wiki it will answer many questions. Remember, Sanchez works without a GUI, and uses command-line arguments which are UNIX-style and either short or long arguments can be used. For example, -b1 --haze 2 -fa
is equivalent to -b 1 -h 2 -f -a
Processing a single image
./Sanchez -s "c:\images\GOES16\**\GOES16_FD_FC_20201004T000018Z.jpg" -o GOES16Output.jpg"
Multiple satellite stitching with auto-crop
./Sanchez reproject -s c:\images -o stitched.jpg --mode stitch -T 2020-08-30T03:50:20 -a
Multiple satellite stitching with auto-crop and timelapse
./Sanchez reproject -s c:\images -o GOESstitched.jpg --mode stitch -I 60 -a
NOTE that in the above example, a Interval of -I 60 is used as relayed IR imagery frequently only comes in every 60 minutes, while direct broadcast comes in every 30 Mins. Sanchez adjust for the gaps in coverage using the Interval argument.
Stitching
When stitching, a target timestamp must be provided. Images in the source directory are scanned and a single image per satellite with the closest timestamp is utilised.
sanchez reproject -o output\stitched.jpg -s sample-images -T 2020-08-30T03:50:20 -a
Satellite Naming Convention
Sanchez has a built in system to recognize the standar naming and timestamp conventions used by both Goestools and XRIT as an example:
GOES16_FD_FC_20201004T070019Z.jpg is NOT the same as GOES16_FD_FC_10042020T070019Z.jpg
Satellite.json file
Sanchez allows you to modify the json file found in the ‘Resource’ Directory to add satellites, or modify them, or, in my case, add a ‘new’ naming convention. Since I use a Custom CLUT, I have add the following to my satellite.json file so that Sanchez recognizes and processes the imagery I save with the custom CLUT.
{
"DisplayName": "GOES-16-FC",
"FilenamePrefix": "GOES16_FD_FC_CUSTOMLUT_",
"Longitude": -75.2,
"LongitudeAdjustment": 0.2,
},
"Brightness": 0.75
},
{
"DisplayName": "GOES-17-FC",
"FilenamePrefix": "GOES17_FD_FC_CUSTOMLUT_",
"Longitude": -137.2,
},
"Brightness": 0.75
},
Adapting Sanchez to work on your sysytem
As I have shown in my other scripts, I use a Windows-based system to process all my satellite imagery. I could use CMD and run Sanchez for a command line, but for me that involves a lot of typing and copying, pasting, etc. So I create a Windows batch file to help automate the scripts I want to run.
Here is an example of a basic Sanchez script that batch processes an image located in F:\SatelliteImagery with a timestamp of 2020-10-04T17:20:00 (within the 30-minute default window) and applies a custom BlueMarble static underlay located in F:\Sanchez\Resources named worldCLUT.200411.3x10848x5424.jpg and outputs the image to and named as F:\Satelliteoutput\GOES16-1710042020stitched.jpg
"F:/Sanchez/Sanchez.exe" reproject -s "F:\SatelliteImagery\*.jpg" stitch -T 2020-10-04T17:20:00 -fa -u "F:\Sanchez\Resources\worldCLUT.200411.3x10848x5424.jpg" -v -o "F:\Satelliteoutput\GOES16-1710042020stitched.jpg" -b 1.2
pause
Creating an animation using Sanchez
I would recommend becoming familiar with the general operation of Sanchez BEFORE adding it to a complete script. That being said, adding Sanchez to a script can result in some very nice imagery results. The script below accomplishes the following task
- Moves the Band IR 13 imagery from my F:\Satellite Imagery\goes16\fd\ch13\ and F:\Satellite Imagery\goes17\fd\ch13\ into one temporary working directory F:\SatelliteIRCH13
- Creates a TimeStamp for file naming.
- Run Sanchez to reproject and stitch both GOES 16 and 17 Imagery into individual batched images, with a static underlay from the folder F:\SatelliteIRCH13\ into a temporary working directory F:\Satelliteoutput\TIMELAPSE
- Resize and Optimize the reprojected imagery from the temporary working directory F:\Satelliteoutput\TIMELAPSE to the temp working directory F:/Satelliteoutput/TIMELAPSE/RESIZED. The file sizes can result in quite large finished mp4, this reduces the sizes while maintaining optimal imagery
- Add my watermark to all the imagery in the temporary working directory F:/Satelliteoutput/TIMELAPSE/RESIZED into the temporary working directory F:/Satelliteoutput/TIMELAPSE/WATERMARKED/
- Create a temporary GIF of all the images from F:/Satelliteoutput/TIMELAPSE/WATERMARKED/ in the temp working directory F:/Satelliteoutput/TIMELAPSE/FINAL/ called 1617EQUIRECTANGULARout.gif
- Create the mp4 from the temp directory of F:/Satelliteoutput/TIMELAPSE/FINAL/ and interpolate and process the gif into a mp4 and save the resultant file with the timestamp created in step 2, into the final file 1617EQUIRECTANGULARout120fp_%month%%day%%year%.mp4
- Delete all of the temporary files.
REM USRADIOGUY timelapse animation code
REM COPY CH13
for /R "F:\Satellite Imagery\goes16\fd\ch13\" %%f in (*.jpg) do copy "%%f" "F:\SatelliteIRCH13"
REM COPY CH13
for /R "F:\Satellite Imagery\goes17\fd\ch13\" %%f in (*.jpg) do copy "%%f" "F:\SatelliteIRCH13"
REM Create TIMESTAMP
set year=%date:~10,4%
set month=%date:~4,2%
set day=%date:~7,2%
set curTimestamp=%month%%day%%year%
REM RUN SANCHEZ TO REPROJECT AND STITCH IMAGERY
"F:/Sanchez/Sanchez.exe" reproject -s "F:\SatelliteIRCH13\*.jpg" stitch -fa -u "F:\Sanchez\Resources\worldCLUT.200411.3x10848x5424.jpg" -v -I 60 -m 2 -o "F:\Satelliteoutput\TIMELAPSE"
echo [101;93m CHECK IMAGERY IN FILE F:/Satelliteoutput\TIMELAPSE\ AND ADJUST IF NEEDED [0m
Pause
REM RESIZE and OPTIMIZE
for /r "F:\Satelliteoutput\TIMELAPSE\" %%a in (*.jpg) do "C:\Program Files\ImageMagick-7.0.9-Q16\magick.exe" mogrify -path ../Satelliteoutput/TIMELAPSE/RESIZED -filter Triangle -define filter:support=2 -thumbnail 1002 -unsharp 0.25x0.25+8+0.065 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB -strip -gravity south -stroke none -pointsize 24 -fill white -font Ailerons -annotate +0+0 GOES_16_17_Channel_13IR "%%~a"
REM ADD WATERMARK
magick mogrify -path "F:/Satelliteoutput/TIMELAPSE/WATERMARKED/" -format jpg -gravity northwest -draw "image over 20,20 0,0 'F:/Satellite Imagery/GOES1617250X50.png'" "F:/Satelliteoutput/TIMELAPSE/RESIZED/*.jpg"
REM CREATE GIF
magick convert -loop 0 -delay 20 "F:\Satelliteoutput\TIMELAPSE\WATERMARKED\stitched*.jpg" "F:/Satelliteoutput/TIMELAPSE/FINAL/1617EQUIRECTANGULARout.gif"
REM CREATE MP4
ffmpeg -i "F:/Satelliteoutput/TIMELAPSE/FINAL/1617EQUIRECTANGULARout.gif" -filter "minterpolate='mi_mode=blend:me_mode=bidir:fps=120:mc_mode=obmc:me=ds:vsbmc=1'" -c:v libx264 -pix_fmt yuv420p "F:/Satelliteoutput/TIMELAPSE/FINAL/1617EQUIRECTANGULARout120fp_%month%%day%%year%.mp4"
pause
del "F:\Satelliteoutput\TIMELAPSE\WATERMARKED\*.*" /s /f /q
del "F:\Satelliteoutput\TIMELAPSE\FINAL\EQUIRECTANGULARout.gif" /s /f /q
del "F:\Satelliteinput\SatelliteIRCH13\*.jpg" /s /f /q
del "F:\Satelliteoutput\TIMELAPSE\REPROJECTED\*.*" /s /f /q
del "F:\Satelliteoutput\TIMELAPSE\RESIZED\*.*" /s /f /q
del "F:\Satelliteoutput\TIMELAPSE\*.jpg" /s /f /q
Updated- 11/23/2020 Sanchez version 1.0.11 Sanchez now has the ability to do full global projections! Here is my full global projection test with the addition of the Satellite ElectroN2, so this one has geostationary satellite imagery from GOES 16 and 17, EWS-G1, Himawari-8, and ElectroN2. 11.23.2020 at 1330-1350 UTC. (some overlays I still need to tighten up) And a demo rendering the full global satellite projection onto a 3d globe as a test, manually turning the 3d globe with mouse control.
Sanchez using the Virtual Satellite mode
When a timestamp or time interval is specified (-t
/ --timestamp
, and -I
/ --interval
, respectively) along with a target longitude (-l
/ --longitude
), Sanchez will perform a multiple satellite stitch and reproject the image back to a geostationary image, simulating a satellite at a given longitude.
This allows for virtual satellite images to be created, comprised of more than one source image.
Longitude sweep
If an end longitude is provided (-E
/ --endlongitude
) along with a timelapse argument, Sanchez will sweep the longitude across the time intervals. By default, the longitude sweep is anticlockwise, however, this may be inverted with --inverse
.
https://github.com/nullpainter/sanchez/wiki/Virtual-satellite
When all this is combined with an animation subroutine, you can create a spinning globe, with accurate projection of the satellite imagery. as shown below:
GOES 16, 17, Himawari8, EWS-G, Elecktro and GK-2A 11.29.2020 1600UTC
GOES 16,17, and Himawari 8 – all imagery came from my sat receiver system.
EWS-G1 from SSEC data site -ssec.wisc.edu, and GK2A from the KMA data site – kma.go.kr, Elecktro from PKC – electro.ntsomz.ru
The base map was made with this data and reprojected using Sanchez by u/nullpainter. This formed the global satellite projection shown here:
This layer was then added to a 3d Globe, this time I used a code from the Google Chrome Experimenters site (called Planet Maker) and added some lines of code to the script that allows the earth to be rotated and then another line that places the moon in orbit around the earth.
Using xplanet to create a day/night underlay for use in Sanchez.
Xplanet has quite a few features, and they can all be used in varying degrees, but, for this demo, we will be just focused on creating a day/night underlay. So start of downloading xplanet and extract it to a folder called xplanet, in my case F:/xplanet. I will be using Windows 10 in my example
All of the images that will comprise your underlay are already installed in the F:/xplanet/xplanet/images folder. You can add to the list with new underlays if you choose. New underlays can be found here: Sanchez-Underlays. But for now we will just stick with the basics.
Once you have extracted the files, create a batch file using notepad, or, better yet Notepad++
Then copy the following into the new batch file:
cd F:\Xplanet echo [101;93m ADJUST THE DATES!!! [0m pause xplanet -body earth -projection rectangular -num_times 1 -geometry 21600x10800 -date 20210211.080000 -output "F:\Sanchez\Resources\xplanet2.jpg" pause
Now what is important here is the -date portion. Let’s say you are using imagery from 02/22/21 at 0800 UTC. you want to make sure the imagery will show the correct day/night underlay at the right date and time, as shown in the code above. If you wanted to change it to say, Dec 3rd, 2020 at 1600UTC then the line would be:
xplanet -body earth -projection rectangular -num_times 1 -geometry 21600x10800 -date 20201203.160000 -output "F:\Sanchez\Resources\xplanet2.jpg"
The next thing to pay attention to is the -geometry tag, you’ll want this to match as closely to your reprojected image. This could be 21600×1080 or 10848×5424 for a lower resolution image. In the example I use, the output image is place into my Sanchez folder at F:/Sanchez/Resources.
Using it in combination with a Sanchez script could be done like this:
set year=%date:~10,4% set month=%date:~4,2% set day=%date:~7,2% set curTimestamp=%month%%day%%year% cd F:\Xplanet echo [101;93m ADJUST THE DATES!!! [0m pause xplanet -body earth -projection rectangular -num_times 1 -geometry 21600x10800 -date 20210212.080000 -output "F:\Sanchez\Resources\xplanet2.jpg" rem DAY and NIGHT SANCHEZ OUTPUT "F:/Sanchez/Sanchez.exe" reproject -s "F:\Satelliteimagelocation\*.*" stitch -T 2021-02-11T08:00:00 -S 0.8 -f -u "F:\Sanchez\Resources\xplanet2.jpg" -v --nocrop -r 2 -o "F:\Satelliteoutput\Your_FILE_NAME_%month%%day%%year%.jpg"
This uses the underlay created by xplanet in Sanchez using multiple satellites and the stitching feature.
Example using the standard full disk.
So, I have been coding and messing around with creating new versions of my scripts for a year or more. The above script has been giving me fits rendering the full video. Turns out I was trying to process too large a file for my system. Reducing the overall projected image from 5424 pixels to 1000 pixels really made a difference. The above image is basically 48 hours of imagery from GOES 16 and 17 (remember I have dual dishes), this imagery is processed to blend false-color imagery from bands 13 and 02 with band 15 (enhanced gradient).
A rough workflow of the scripting is this.
- Derive times using wmic to get current UTC and Local times for variables in Script
- Ask for input from user for the start date to recall imagery from by MM-DD-YYYY
- Ask for input from user for the start date for Sanchez to run reprojection from from by YYYY-MM-DD T HH:MM:SS
- Copy all selected files based on input date (from step 2.) to temporary folders.
- Resize all GOES 16 FC Imagery, optimize and copy to another temp folder
- Resize all GOES 16 IR Imagery, optimize, and render transparent and copy to another temp folder
- Resize all GOES 17 Imagery, optimize and copy to another temp folder
- Resize all GOES 17 IR Imagery, optimize, and render transparent and copy to another temp folder
- Manually check file count of each sub folder to make sure it is the same ie. 179 files in false color, and 179 in IR. (working on a way to do this via automated scripting).
- Using a delayed expansion method, create GOES 16 file lists A and B, Then use ImageMagick -compose blend command to layer the images.
- Using a delayed expansion method, create GOES 17 file lists A and B, Then use ImageMagick -compose blend command to layer the images.
- Move all composited imagery from GOES 16 and 17 into a singel temp folder.
- Use Sanchez to reproject the imagery into a equirectangular projection using the time stamp variable in step 3.
- Resize the imagery to a height of 1000 pixels regardless of the width using Image Magick mogrify and -resize commands
- Add my “USRADIOGUY.COM” logo watermark to every image.
- Again, using delayed expansion, renumber all imagery so that it can be combined in step 18.
- Create Timestamp varibles for mp4 file creation.
- Use FFMPEG to generate and interpolated (using the minterpolate function) mp4 of a certain framerate.
- Choose to enable a subroutine to add a music track to final videos and/or create a video that is compatible for twitter.
- Delete all temporary file from folders while leaving folder structure intact.
- ENDLOCAL to clear variables form system memory.
- Exit the batch file
Adding Specular Lighting and Rayleigh Scattering to imagery.
First off, what is Rayleigh Scattering?? You might have wondered why the sky is usually blue, yet red at sunset. The optical phenomenon which is (mostly) responsible for that is called Rayleigh scattering. When looking at imagery of the earth from space, especially from low earth orbit, one notices the atmosphere tinged in blue as the light passes through it to the observer’s eye. Since GEO sats are about 23,000 miles away, and the visible atmosphere is so thin, it becomes invisible to the spacecraft. I have found a method to replace that in full disk imagery from global GEO sats.
Additionally, I have added specular lighting to the earth imagery to represent the sunlight ‘glinting’ off of water (lakes and oceans).
Blue Sky ~Rayleigh Scattering
The interaction between light and matter is extremely complex, and there is no easy way to fully describe it. Modeling atmospheric scattering is, in fact, exceptionally difficult. Part of the problem comes from the fact that the atmosphere is not a homogeneous medium. Both its density and composition change significantly as a function of the altitude, making it virtually impossible to come up with a “perfect” model.
From a logical point of view, it makes sense that the strength of the scattering is proportional to the atmospheric density. More molecules per square meter mean more chances for photons to be scattered. The challenge is that the composition of the atmosphere is very complex, consisting of several layers with different pressures, densities, and temperatures. Luckily enough, most of the Rayleigh scattering happens in the first 60Km of the atmosphere. Within the troposphere, the temperature decreases linearly and the pressure decreases exponentially.
The blue color of the sky is caused by the scattering of sunlight off the molecules of the atmosphere. This scattering, called Rayleigh scattering, is more effective at short wavelengths (the blue end of the visible spectrum). Therefore the light scattered down to the earth at a large angle with respect to the direction of the sun’s light is predominantly in the blue end of the spectrum.
Note that the blue of the sky is more saturated when you look further from the sun. The almost white scattering near the sun can be attributed to Mie scattering, which is not very wavelength-dependent.
The water droplets that make up the cloud are much larger than the molecules of the air and the scattering from them is almost independent of wavelength in the visible range.
I have used a combination of software tools to achieve this phenomenon.
My basic workflow to achieve this:
- Download from my local satellite recievers:
- GOES 16 Band 13
- GOES 17 Band 13
- Himawari 8 IR Imagery
- Download from remote satellite servers:
- GK2A IR Imagery
- GOES 13 (EWS-G1) IR imagery
- Electro L2
- Compile and stitch the imagery into a global composite using Sanchez to stitch the global composite.
- Using Xplanet software:
- Create Rayleigh scattering lookup tables. Tables are calculated for fixed phase angles (the observer – point – sun angle). For lines of sight intersecting the disk, the tables are a 2D array of scattering intensities tabulated as a function of incidence (the zenith angle of the sun seen from the point of interest) and emission (the zenith angle of the observer seen from the point of interest).
- Specify Both Rayleigh Lighting and Specular Lighting in the config files for Xplanet.
- Using a subroutine in the batch file, I automatically generate the config file that xplanet need to generate the imagery based on the files generated by Sanchez.
- Run Xplanet with the selected config to generate multiple planet views from each of the satellites longitude (ie GOES 16, -75.2 or GOES 13, 65) (Global Images below)
- If desired download the latest TLE’s for the selected satellites to place the orbits on the map (in this case I used a few L.E.O. weather sats
- If desired select and use the Yale Bright Star Catalog to display, and name Stars.
- Generate a Full Global equirectangular image (Image 3)
GOES Full-Disc Animations in Linux, and spaghetti sauce!
Matt Gibson has created a tutorial on how to animate the GOES full-disc images using Linux as both the client and host. This will allow you to generate the animations on the Linux system that your images are stored on, and then transfer them to your system for viewing, sharing, etc.
Here is the link to Matt’s tutorial https://mattgibson.ca/generate-goes-full-disc-animations-in-linux/
And like most of us, Matt has other interests as well, so check out his other items on his blog. In between articles such as “Setting up a Matrix server for Secure Encrypted Communications” and “Making wxtoimg work on your 64-bit arm Odroid N2+“, you might just find a great recipe for spaghetti sauce!