It's possible to save RAW files (mostly) unprocessed with iPhones, either via built-in functionality (Pros) or via apps like Halide.
But the aggressiveness of the de-noising in the native JPG/HEIF images otherwise is really unfortunate if you want to look at the images on a screen larger than the phone's screen. The amount of detail lost (other than in areas like people's faces where the phone knows to specialise) can be very considerable.
I'd really like a way to dial that aggressiveness down a fair bit, even at the cost of more noise/grain and larger file size (through less compression due to the extra noise).
Another thing is the amount of lens flare you can get when shooting at the sun for sunsets/rises, etc or other large bright light sources. With very small lens elements, from a physics perspective it's understandable that suppressing the reflections and inter-reflections is very difficult on such a small surface area (even with special coatings to reduce the fresnel reflection ratios), but if you care about image quality and wanting to look at images on screen larger than the phone which took them, larger format cameras still have some benefit despite their larger and heavier size and therefore inconvenience (looks at 5D Mk IV on shelf).
I wish there was a middle ground between what Android/Pixel camera saves as raw, and the in-camera JPEG. Sometimes I have a few quibbles with the JPEG and what I'd like to do is edit the raw file, but starting from something close to the JPEG. Unfortunately what you get as a starting point from raw is hideous, and it's never clear how to begin. I don't think I've ever got an acceptable result trying to edit raw photos from my Pixel.
> and it's never clear how to begin
Start by adjusting the black levels and the exposure. This is where the histogram can help to visualize how much you are adjusting (if you have one). As the exposure goes up, you can adjust the highlights to recover some of the areas trying to blow out. As you pull the black level down, you can recover some of the details getting crushed using with the shadows adjustment. You can then adjust the contrast/saturation/warmth/tint as needed. The order of adjustments in the iOS Photos editor pretty much follow that order.
In other words, you want either your camera app to select the initial tweaks for you to be able continue in the external editor (not going to happen, RAW editing software is incompatible by design), or your editing software to select the initial tweaks that "look good" (that depends on your software). In RAW mode, Google Camera's output is photometrically correct, even if it stacks multiple frames or denoises it. Which is the only way to do it that makes sense, any other RAW camera app or actual dedicated camera does this the same way.
Or you could provide the RAW and the JPEG and it would start you off at a point that most closely matches the JPEG?
That's exactly what I'm talking about, how do you imagine that working? Metadata is not compatible by design, because processing pipelines are all subtly different and your result will always look different in your editor. Trying to match some basic parameters with the JPEG is possible and some RAW software can do that, but the result is going to be subtly different for the same reason.
Search over all values of all parameters and choose the ones the minimise the mean square error of pixel values.
Obviously that will be slow, so probably do some kind of gradient descent, or perhaps depending on what the parameters are there may be a closed-form solution, I don't know.
Yes the result will be subtly different but it's just a starting point.
Isn't that what the "auto" button does, and then you can tweak from there?
I've never had to write such software but in my imagination there is the sensor data, potentially from several exposures, and some static data about the camera, and a list of edits and parameters that the photo app is using to produce the in-camera JPEG. And I just want a way to intervene in that list of edits and parameters to produce my own result. There must be SOME way to do this otherwise how do I edit raws from my real camera? The starting point for camera raw in my photo editor always looks great if the file came from my camera, and always looks ghastly if it came from my mobile.
For Android, you can sort of get some of this with Snapseed. I occasionally use it, and it's "ok". I'm more frustrated by the fact that my preferred RAW editor (DxO) doesn't handle Android's DNG files. For me, at least, editing raw images on a phone screen is just not tolerable.
Do you know Lumina [1]? It has a setting that turns off most post-processing, resulting in very natural-looking pictures. It has more features than I need, and it's free.
[1] https://apps.apple.com/us/app/lumina-manual-camera/id1617117...
Ooh, this looks really good. I use OpenCamera on Android but it's pretty limited... I wonder if there's anything like Lumina there.
Wow, this is a nice camera app. I usually stick with the stock iOS camera but this is a tangible upgrade. Generous to offer it for free.
In our camera app Photon (https://photon.cam) we have an option "Process Control" to blend the RAW and JPEG, bringing some of the noise from the RAW back into the JPEG but keeping the better dynamic range from Apples HDR algorithms
It's strange that in the age of AI, denoisers are still so bad. It's basically impossible to photograph snowing in the winter because the denoiser will remove 90% of the snowflakes. Machine learning models are already used for denoising ray traced graphics with substantially improved results, so why is it that cameras aren't using ML denoisers yet? At least for still images. Or do they perhaps already use them, only the quality is still bad for unknown reasons?
(As someone who worked closely with pathtracing renderers and de-noisers, I think I can answer this :) )
It's mostly because in the VFX/CG space for ray tracing/path tracing de-noisers, they almost always rely on extra outputs/AOVs of things like 'albedo' (diffuse reflectance), normal / world position, etc, to help guide them in many cases.
So they often can 'cheat' a bit, and know where the edges of things are (because say the object ID AOV changes - minus pixel filtering, which complicates things a bit).
They can also 'cheat' in other ways, by mixing back in some of the diffuse texture detail that the denoiser might have removed from the 'albedo' AOV channel.
Cameras don't really have anything to guide them, so they have to guess. And often, they seem to use very primitive methods like bi-lateral filters (or at least things which look very similar), to try and guide them, but it doesn't work very well.
Portrait cameras on phones can use depth sensors a bit to help if the camera has them, but for things like hair strands, it doesn't really work, and is mostly useful for fake-depth-of-field depth-based blurring.
Yeah, but surely ML models would at least work better than analytic algorithms. After all, when looking at a noisy picture, our brain is pretty good at distinguishing detail from noise, so it's not clear to me why an ML model couldn't have denoising performance similar to the human brain, even if it doesn't match the "cheating" denoisers used in ray tracing.
It would probably help you to compare what you can do on a phone vs what you can do with desktop software (Lightroom/Photoshop, DxO, Topaz, CaptureOne, etc). It's generally quite good, with the exception of challenging liminal areas (e.g. hair, foliage).
Fwiw, Topaz -- which I have a license for but essentially never use -- has pretty incredible denoising & upsizing features (for both photo & video), but to get the optimal quality output you offload the processing to their cloud infra (and buy credits from them to pay for it). It's roughly the equivalent of a SWE using a local LLM that's good enough" vs a frontier model that's SOTA but requires a consumption-based subscription.
Interesting, so it seems to be an issue with heavy compute or RAM requirements.
I've got a 61mp camera, and an RX 7900XT. It takes about 15s/picture for DXO to denoise, which is a lot longer than people are willing to wait on a phone to take a photo. Topaz is even slower. A cloud service could be used to do it in post, but someone has to pay for that.
Yet in modern computer games, modern graphics cards denoise a scene in real-time at 60 frames per second using machine learning models [1][2] while doing all the other rendering at the same time. Granted, that's ray tracing, and the resolution is lower, and they technically cheat by using additional information, but it might be that DXO is not optimized very well.
1: https://blogs.nvidia.com/blog/ai-decoded-ray-reconstruction/
2: https://gpuopen.com/amd-fsr-rayregeneration/
Games will typically un at 4k or less which is about 8mp on the other hand it’s difficult to buy a stills camera with less than 20 mp, and >40mp is common. Most algorithms are n^2 in graphics as well so we wouldn’t expect a linear speed up. I’ve tried dxo, Lightroom, and topaz they all perform about the same so I don’t think it’s particularly unoptimized.
Would you be happy if ML detects that you are trying to capture the moon and replaced your shot with a memorized picture of the moon with match ing size and orientation ?
Autoencoders can presumably do this except that it would be operating at levels of patches rather than the entire Moon.
No would be unhappy :(
ML models on some Samsung phones did exactly that.
https://news.ycombinator.com/item?id=35107601
It was probably a result of an explicit company charter but such things can happen with modern ML models quite unintentionally.
> Yeah, but surely ML models would at least work better than analytic algorithms.
Bear in mind, a modern flagship phone doesn't just need to take photos - it also needs to record 4k video at 60fps.
Can't go too hard with the ML when you've only got 1/60th of a second to do it.
I don't think this is even just an AI denoiser problem. Rain effectively looks like haze for the parts of the scene that are at sufficiently long distances. Therefore it's difficult to use dehaze with rainy scenes. The harder the rain is falling, the more limited your ability to dehaze. It's much the same with denoise. You won't be denoising at 100% in Topaz if it's dark and raining hard.
One day smartphone cameras might get there, but right now, the sensor technology isn't there yet. The problem isn't merely noise. It's that rain and snow are moving about in the scene, which means that the camera can't just do its usual trick of taking multiple exposures.
AI denoisers are nice, but they aren't strictly necessary for ILCs. Even those full-frame cameras without IBIS are able to take pictures of nighttime snow just fine.
Are we still talking about smartphone cameras? If yes, apps already heavily rely on much more advanced computational photography than your average photo editor can do, including but not limited to ML denoisers. The problem is that such apps are typically optimized for the "average case" and are as automated as possible, so they either remove snow, rain, and haze intentionally, or lose small moving particles as the result of stacking. That said, snow and rain are usually possible to capture in the apps that attempt to determine the scene type or have specific modes.
Would you have an example of an app which can photograph snow and rain?