The following is a step by step process for modifying libphoto and FlashPhoto to add a new filter. In these instructions, <Filter> will be a stand in for the name of the filter you are adding.
First, we need to change our libphoto files to implement the new filter. Open the src/ folder in the libphoto/ directory. This folder contains the source code for the libphoto library, and is where you will put the bulk of your changes. The changes you'll need to make will be slightly different based on whether or not you're adding a convolution filter or not.
Convolution Filter:
-Enums.h:
-Add FILTER_<Filter> to the filter_enum enumerated type. Add any parameters your filter users to the filter_parameters struct.
-MaskFactory.cpp, MaskFactory.h:
-Add a private static mask function called get<Filter>Kernel() that creates and returns a mask with the kernel values appropriate to the filter you'd like to create. If the kernel will be variable depending on some parameters, you can pass those into the function. You can use the other getMask() functions in the file as a reference.
-Add get<Filter>Kernel() to the .h file.
-Modify the switch statement in MaskFactory::getKernel() to call the get<Filter>Kernel() function you wrote.
-Filter.cpp, Filter.h:
-Write apply<Filter>Filter(PixelBuffer*, filter_parameters). This function calls MaskFactory::getKernel() to retrieve the kernel, and passes it to the convolve function along with the PixelBuffer* to be modified. FilterMethods::convolve() takes care of the hard part.
-Add the function you just wrote to Filter.h as a static public function.
Non-Convolution Filter:
-Filter.cpp, Filter.h:
-Write your static public apply<filter>Filter(PixelBuffer, [parameters]) in Filter.cpp. Add it to Filter.h.
After writing your filters you'll need to update the library file in libphoto. Navigate to the base libphoto directory and run "make clean; make"
Now you'll actually want to change FlashPhoto to allow the program to use the filter you just wrote. First, navigate to FlashPhoto/src/.
Modify FlashPhoto.cpp, FlashPhoto.h:
-Change FlashPhoto::initGlui() to initialize the glui elements for the new filter. This should be similar to how the other filters are initialized, so shouldn't be too hard to figure out.
-Add a UI_APPLY_<FILTER> value to the UIControlType in the .h file
-In the "glui callbacks" section of the .cpp file, add a FlashPhotoApp::applyFilter<Filter>() function that takes no arguments, calls the appropriate filter function in Filter.cpp, and saves the pixelbuffer. Again, this should be very similar to the other filter functions present already. Don't forget to add it to the .h file as well.
-In FlashPhotoApp::gluiControl(), add a case to the switch statement for UI_APPLY_<FILTER> that calls the applyFilter<Filter>() function you just wrote.
Recompile FlashPhoto by going to the main directory for it and running make clean; make. If you've done everything right, your new filter should show up in the menu, ready to use.