media: i2c: Add driver for Sony IMX500 sensor

The Sony IMX500 is a stacked 1/2.3-inch CMOS digital image sensor and
inbuilt AI processor with an active array CNN (Convolutional Neural
Network) inference engine.  The native sensor size is 4056H x 3040V, and
the module also contains an in-built ISP for the CNN. The module is
programmable through an I2C interface with firmware and neural network
uploads being made over SPI. This driver supports imaging only.

Signed-off-by: Richard Oliver <richard.oliver@raspberrypi.com>

media: i2c: imx500: Inbuilt AI processor support

Add support for the IMX500's inbuilt AI processor. The IMX500 program
loader, AI processor firmware, DNN weights are accessed via the kernel's
firmware interface on 'open' and are transferred to the IMX500 over SPI.

Signed-off-by: Richard Oliver <richard.oliver@raspberrypi.com>

media: i2c: imx500: Enable LED during SPI transfers

The Raspberry Pi 'AI Camera' is equipped with an LED. Enable this LED
during SPI transfers to indicate to the end-user that progress is being
made during large tramsfers.

Signed-off-by: Richard Oliver <richard.oliver@raspberrypi.com>

drivers: media: imx500: Fixes for vblank control

Reduce the default/max framerate of the 2x2 binned mode to 30fps.
The current limit of 50fps can cause the sensor to produce corrupt
frames and cause missing framing events.

Also fixup the vblank control min/max/default/step paramters when
setting up.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>

drivers: media: imx500: Simplify the vblank control init

Set the VBLANK control minimum and default values to IMX500_VBLANK_MIN
unconditionally everywhere.

Remove the mode specific framerate_default parameter, it is now unused.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>

drivers: media: imx500: Enable LS correction

This correction is calibrated to approx 5000K.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>

imx500: Fix for long exposure setup

The IMX500 (unlike the IMX477/IMX708) requires two regsiters to be set
for the exposure shift value to work correctly. The additional register
write (which was missing) is for the integration time shift.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>

drivers: media: imx500: Enable sensor temperature monitoring

The register needs to be disabled before loading any firmware, otherwise
the upload fails for unknown reasons. Re-enable before starting the
sensor streaming.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>

drivers: media: imx500: Add device id readback control

Add a new custom control V4L2_CID_USER_GET_IMX500_DEVICE_ID to allow
userland to query the device id from the IMX500 sensor eeprom.

Note that this device id can only be accessed when a network firmware
has been upoloaded to the device, so cannot be cached on probe.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>

media: i2c: imx500: pm_runtime error paths

This change amends various error-paths in imx500_start_streaming() to
ensure that pm_runtime refcounts do not remain erroneously incremented
on failure.

Signed-off-by: Richard Oliver <richard.oliver@raspberrypi.com>

media: i2c: imx500: GPIO acquire/release semantics

When the imx500 driver is used as part of the 'AI Camera', the poweroff
state is never reached as the camera and gpio driver share a regulator.
By releasing the GPIOs when they are not in use, 'AI Camera' is able to
achieve a powered-down state.

Signed-off-by: Richard Oliver <richard.oliver@raspberrypi.com>

media: i2c: imx500: input tensor injection

Input tensor injection is a debug feature that allows a user-controlled
input to be passed directly to IMX500's inference engine (bypassing the
in-built ISP).

Three new custom controls are added to ENABLE_INJECTION before streaming
begins, to provide appropriate input tensors via an INPUT_TENSOR_FD, and
to provide notification of DNN results in the sensor output via
INJECTION_CMP_FRM.

Signed-off-by: Richard Oliver <richard.oliver@raspberrypi.com>

MAINTAINERS

@@ -24075,6 +24075,7 @@ L:	linux-media@vger.kernel.org
 S:	Maintained
 T:	git git://linuxtv.org/media_tree.git
 F:	Documentation/devicetree/bindings/media/i2c/sony,imx500.yaml
+F:	drivers/media/i2c/imx500.c
 
 SONY IMX519 SENSOR DRIVER
 M:	Arducam Kernel Maintenance <info@arducam.com>

drivers/media/i2c/Kconfig

@@ -313,6 +313,18 @@ config VIDEO_IMX477
 	  To compile this driver as a module, choose M here: the
 	  module will be called imx477.
 
+config VIDEO_IMX500
+	tristate "Sony IMX500 sensor support"
+	depends on I2C && VIDEO_DEV
+	select VIDEO_V4L2_SUBDEV_API
+	select V4L2_CCI_I2C
+	help
+	  This is a Video4Linux2 sensor driver for the Sony
+	  IMX500 camera.
+
+	  To compile this driver as a module, choose M here: the
+	  module will be called IMX500.
+
 config VIDEO_IMX519
 	tristate "Arducam IMX519 sensor support"
 	depends on I2C && VIDEO_DEV

drivers/media/i2c/Makefile

@@ -65,6 +65,7 @@ obj-$(CONFIG_VIDEO_IMX355) += imx355.o
 obj-$(CONFIG_VIDEO_IMX412) += imx412.o
 obj-$(CONFIG_VIDEO_IMX415) += imx415.o
 obj-$(CONFIG_VIDEO_IMX477) += imx477.o
+obj-$(CONFIG_VIDEO_IMX500) += imx500.o
 obj-$(CONFIG_VIDEO_IMX519) += imx519.o
 obj-$(CONFIG_VIDEO_IMX708) += imx708.o
 obj-$(CONFIG_VIDEO_IR_I2C) += ir-kbd-i2c.o

include/uapi/linux/v4l2-controls.h

@@ -232,6 +232,12 @@ enum v4l2_colorfx {
  * We reserve 16 controls for this driver. */
 #define V4L2_CID_USER_BCM2835_ISP_BASE		(V4L2_CID_USER_BASE + 0x10e0)
 
+/*
+ * The base for IMX500 driver controls.
+ * We reserve 16 controls for this driver.
+ */
+#define V4L2_CID_USER_IMX500_BASE		(V4L2_CID_USER_BASE + 0x2000)
+
 /* MPEG-class control IDs */
 /* The MPEG controls are applicable to all codec controls
  * and the 'MPEG' part of the define is historical */