Friday, December 21, 2012

Sobel Filter Application on the Xilinx Zynq Zedboard

Here are the steps for getting the Sobel Filter application running on the Zynq Zedboard using a webcam for the input data stream while the output frame is shown in the HDMI display. The Sobel Filter is implemented as a hardware accelerator on the fpga as well as an openCV software implementation on the arm processor.



Download Link:
Git Link:
UBoot File used:

Steps to get the application running on the board:

1. Connect Power, HDMI display, USB HUB with keyboard, mouse and webcam (I used a Logitech webcam) to the Zedboard.

2. Format the SD Card using the instructions as given in the link below:
http://wiki.analog.com/resources/tools-software/linux-drivers/platforms/zynq#enable_xf86-video-modesetting_xorg_driver
The first partition needs to have the  boot.bin,  zImage,  and dts files. These can be found on the SD_card folder in the zip file. The second partition needs to have the Linaro file system (also found at the above link).

3. Set mode on the Zedboard to SD card boot mode.

4. Power on and boot into linux.

5. Install OpenCv by using the command "sudo apt-get install libopencv-dev" (sudo password is linaro).

6. Connect an ethernet cable and copy folder test_app in the zipfile to the filesystem.
(cf_adv7511_zed\workspace\test_app).

7. Run command "cmake ."

8. Run "make"

9. Run "sudo ./camera"

10. It will display the original webcam stream and you can enable hardware sobel filtering and software sobel filtering using "h" and "s" respectively. Original stream can be enabled with "o". Press escape to exit anytime.

Note. This design is not fully optimized  as there is a significant communication bottleneck. This bottleneck is caused while copying the video stream buffer to the DMA region and back, as well as due to the slow latency of the DDR. For the first problem, I am looking into upgrading the kernel to the 3.8 version to use the DMA buffer sharing options. For the second problem, I'm working on using the Accelerator Coherency Port, as it provides a low latency path directly from the cache to the accelerator.

An EDK project is also shared with a bare metal application to test the hardware without the linux kernel.

This reference design was made available from the PARSE Research group at TU/e.

References-
http://wiki.analog.com/resources/tools-software/linux-drivers/platforms/zynq#enable_xf86-video-modesetting_xorg_driver
http://wiki.xilinx.com/zynq-base-trd-14-3
http://ez.analog.com/message/70323#70323

27 comments:

  1. Hi,

    Few questions about your release.
    Does your branch come from Xilinx TRD 14.3?
    Which DTS and defconfig files do I use to build your code?

    andy

    ReplyDelete
  2. Hi,

    Does it modified from Xilinx TRD 14.3?
    Which defconfig and dts files should I use?

    andy

    ReplyDelete
  3. My branch is shared on the git link above which contains all the files dts(zynq-zed-sobel) and defconfig(zync_basic0_sd_defconfig) files. It is modified from analog/digilent linux kernel tree, which was a newer version than the xilinx kernel tree when I started it.

    ReplyDelete
  4. Current git is updated now:
    dts:
    https://github.com/shakith/zedboard/blob/master/arch/arm/boot/dts/zynq-zed-basic.dts

    defconfig: https://github.com/shakith/zedboard/blob/master/arch/arm/configs/zync_basic0_sd_defconfig

    ReplyDelete
  5. Hi Fernando,

    Firstly thank you for your sharing about sobel filter on linarıo system. But I can not install the opencv-dev. below is my log. please can you help me?

    linaro@linaro-ubuntu-desktop:~# sudo apt-get install opencv-dev
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    E: Unable to locate package opencv-dev

    Thank you again and best regards.

    Kursat Gol

    ReplyDelete
  6. Hi Fernando,

    I solve my last problem changing this. The command should be like this: 'sudo apt-get install libopencv-dev' But I have a problem too again. Below commands are not running I think there is a problem on 'test_app' directory. Where to should I extract this 'test_app' directory? What are the right commands to 'make' the camera application. I use a Piranha webcam. Maybe can this be a problem? If I have use a Logitech webcam as you mentioned? Please help me Fernando.

    7. Run command "cmake ."
    8. Run "make"
    9. Run "sudo ./camera"

    Thanks in advance. Best regards.

    Kursat Gol

    ReplyDelete
  7. Good. you found the new opencv package.
    You can put in test_app anywhere with the right permissions. e.g. your home folder.
    Is the webcam detected and tested? https://help.ubuntu.com/community/Webcam

    ReplyDelete
  8. cheese is also not running too. I am going to buy a Logitech webcam now. :) I have one more question too. Shall I run these (cmake.-make-./camera) command in the 'test_app' directory or anywhere?

    thank you again for your kindly support.

    Kursad Gol

    ReplyDelete
  9. my error log is here:

    linaro@linaro-ubuntu-desktop:~\test_app$ cmake.
    cmake.: command not found

    linaro@linaro-ubuntu-desktop:~\test_app$ make
    make: /usr/bin/cmake: Command not found
    make: *** [cmake_check_build_system] Error 127

    Thank you Fernando.

    ReplyDelete
  10. Yep. In the same directory.
    sudo apt-get install cmake or similar :)

    ReplyDelete
  11. Hi Fernando,

    Sorry for inconvenience but your code is not running on my zedboard. everything seems ok but I can not install the cmake. Below is my error chain. Please advice me something. Thank you very much indeed. Now I have a new problem like gstreamer0.10-plugins-bad_0.10.23-1

    linaro@linaro-ubuntu-desktop:~/test_app$ sudo apt-get install cmake
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    You might want to run 'apt-get -f install' to correct these:
    The following packages have unmet dependencies:
    cmake : Depends: libxmlrpc-core-c3 but it is not going to be installed
    Depends: cmake-data (= 2.8.7-0ubuntu5) but it is not going to be installed
    gnome-video-effects : Depends: gstreamer0.10-plugins-bad but it is not going to be installed
    E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
    linaro@linaro-ubuntu-desktop:~/test_app$ sudo apt-get -f install
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Correcting dependencies... Done
    The following extra packages will be installed:
    gstreamer0.10-plugins-bad
    Suggested packages:
    frei0r-plugins
    The following NEW packages will be installed:
    gstreamer0.10-plugins-bad
    0 upgraded, 1 newly installed, 0 to remove and 421 not upgraded.
    2 not fully installed or removed.
    Need to get 0 B/1936 kB of archives.
    After this operation, 4541 kB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    (Reading database ... 116988 files and directories currently installed.)
    Unpacking gstreamer0.10-plugins-bad (from .../gstreamer0.10-plugins-bad_0.10.23-1+ti1.6.5.3+1linaro1_armhf.deb) ...
    dpkg: error processing /var/cache/apt/archives/gstreamer0.10-plugins-bad_0.10.23-1+ti1.6.5.3+1linaro1_armhf.deb (--unpack):
    trying to overwrite '/usr/lib/arm-linux-gnueabihf/gstreamer-0.10/libgstcamerabin2.so', which is also in package gstreamer0.10-plugins-good 0.10.31-1ubuntu2~linaro1
    dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
    Processing triggers for libglib2.0-0 ...
    Errors were encountered while processing:
    /var/cache/apt/archives/gstreamer0.10-plugins-bad_0.10.23-1+ti1.6.5.3+1linaro1_armhf.deb
    E: Sub-process /usr/bin/dpkg returned an error code (1)

    Best regards.

    Kursat Gol

    ReplyDelete
    Replies
    1. Hi,

      I'm getting the same error while installing cheese- in order to detect the functionality of my webcam. Any suggestions?

      Delete
  12. Hi again Fernando. I am trying since yesterday morning all of the day and night Friday like all Electronic in love guys but I think there is something wrong explained on your blog page. I bought a Logitech C270 webcam and connect to Zedboard USB OTG port. I installed the openlibcv-dev package. Also I installed 'sudo apt-get install cmake' package too. But still I have errors. Below is my error logs. What am I doing wrong or which issue I can not understand? Please could you help me clearly? Thank you very much indeed.

    linaro@linaro-ubuntu-desktop:~$ cd test_app/
    linaro@linaro-ubuntu-desktop:~/test_app$ sudo make
    make[2]: *** No rule to make target `CMakeFiles/camera.dir/build'. Stop.
    make[1]: *** [CMakeFiles/camera.dir/all] Error 2
    make: *** [all] Error 2
    linaro@linaro-ubuntu-desktop:~/test_app$ sudo cmake CMakeFiles/camera.dir/build.make
    CMake Error: The source directory "/home/linaro/test_app/CMakeFiles/camera.dir/build.make" is a file, not a directory.
    Specify --help for usage, or press the help button on the CMake GUI.
    linaro@linaro-ubuntu-desktop:~/test_app$ sudo cmake /home/linaro/test_app/
    -- Configuring done
    CMake Error: CMake can not determine linker language for target:camera
    CMake Error: Cannot determine link language for target "camera".
    -- Generating done
    -- Build files have been written to: /home/linaro/test_app
    linaro@linaro-ubuntu-desktop:~/test_app$

    ReplyDelete
    Replies
    1. Hi,

      Did you solve this problem?

      Delete
    2. He managed to fix it. Drop me an email with the exact problem?

      Delete
  13. Hi Fernando.
    Thanks for the design. I tried to boot linux on my zedboard. But it is not able to boot it properly. Below is the log. Please provide your comments.
    Thanks in advance.

    [drm] Initialized drm 1.1.0 20060810
    brd: module loaded
    loop: module loaded
    xqspips e000d000.spi: master is unqueued, this is deprecated
    xqspips e000d000.spi: at 0xE000D000 mapped to 0xDF80E000, irq=51
    libphy: XEMACPS mii bus: probed
    xemacps e000b000.eth: pdev->id -1, baseaddr 0xe000b000, irq 54
    ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
    usb_hcd_xusbps_probe: No OTG assigned!
    usb_hcd_xusbps_probe: OTG now assigned!
    xusbps-ehci xusbps-ehci.0: Xilinx PS USB EHCI Host Controller
    xusbps-ehci xusbps-ehci.0: new USB bus registered, assigned bus number 1
    xusbps-ehci xusbps-ehci.0: irq 53, io mem 0x00000000
    xusbps-ehci xusbps-ehci.0: USB 2.0 started, EHCI 1.00
    hub 1-0:1.0: USB hub found
    hub 1-0:1.0: 1 port detected
    Initializing USB Mass Storage driver...
    usbcore: registered new interface driver usb-storage
    USB Mass Storage support registered.
    mousedev: PS/2 mouse device common for all mice
    i2c /dev entries driver
    usbcore: registered new interface driver uvcvideo
    USB Video Class driver (1.1.1)
    sdhci: Secure Digital Host Controller Interface driver
    sdhci: Copyright(c) Pierre Ossman
    sdhci-pltfm: SDHCI platform and OF driver helper
    mmc0: Invalid maximum block size, assuming 512 bytes
    No connectors reported connected with modes
    [drm] Cannot find any crtc or sizes - going 1024x768
    mmc0: SDHCI controller on e0100000.sdhci [e0100000.sdhci] using ADMA
    Console: switching to colour frame buffer device 128x48
    fb0: frame buffer device
    drm: registered panic notifier
    [drm] Initialized analog_drm 1.0.0 20110530 on minor 0
    usbcore: registered new interface driver usbhid
    mmc0: new high speed SDHC card at address 1234
    usbhid: USB HID core driver
    mmcblk0: mmc0:1234 SA04G 3.63 GiB
    TCP: cubic registered
    NET: Registered protocol family 17
    mmcblk0: p1 p2
    VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
    Registering SWP/SWPB emulation handler
    drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
    ALSA device list:
    No soundcards found.
    Waiting 1sec before mounting root device...
    EXT4-fs (mmcblk0p2): recovery complete
    EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
    VFS: Mounted root (ext4 filesystem) on device 179:2.
    Freeing init memory: 152K
    Failed to execute /sbin/init. Attempting defaults...
    Kernel panic - not syncing: No init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
    [] (unwind_backtrace+0x0/0xe0) from [] (panic+0x88/0x1e0)
    [] (panic+0x88/0x1e0) from [] (init_post+0xa4/0xc8)
    [] (init_post+0xa4/0xc8) from [] (kernel_init+0x17c/0x1b8)
    CPU1: stopping
    [] (unwind_backtrace+0x0/0xe0) from [] (ipi_cpu_stop+0x3c/0x6c)
    [] (ipi_cpu_stop+0x3c/0x6c) from [] (handle_IPI+0x68/0xb4)
    [] (handle_IPI+0x68/0xb4) from [] (gic_handle_irq+0x54/0x5c)
    [] (gic_handle_irq+0x54/0x5c) from [] (__irq_svc+0x44/0x78)
    Exception stack(0xdc475fa0 to 0xdc475fe8)
    5fa0: dc474000 0000b174 dc475fd8 dc46c0c0 1c46006a 00000015 10c0387d c0546cd4
    5fc0: 0000406a 413fc090 00000000 00000000 00000000 dc475fe8 dc474018 c000e36c
    5fe0: 20000013 ffffffff
    [] (__irq_svc+0x44/0x78) from [] (default_idle+0x28/0x30)
    [] (default_idle+0x28/0x30) from [] (cpu_idle+0x8c/0xe4)
    [] (cpu_idle+0x8c/0xe4) from [<0035b094>] (0x35b094)
    panic occurred, switching back to text console

    ReplyDelete
  14. It couldnt find init. I would check the 2nd partition for the following things? format, active, enough size and the rootfs installed.

    ReplyDelete
  15. I was able to boot using the sd card contains you provided but I am not able to find the u-boot.elf required for the generating the boot.bin. I found the fsbl.elf and system.bit. Can you provide the u-boot.elf required for building boot.bin.with regards,
    amit salaskar

    ReplyDelete
  16. Hi Fernando,

    Now its working fine. There was a problem in the 2nd partition.

    Observation: In the demo, SW filtering is taking less time (around 10 ms per frame) than HW filtering (around 90-100 ms per frame). Is it because of communication bottleneck of AXI?

    I have some doubts. Please provide your comments on following:
    1. Are you using AXI stream or AXI memory mapped burst mode?
    2. Is their any way to directly stream the video data to FPGA fabric without intervention of CPU?

    Thanks

    ReplyDelete
  17. hi abhishek,
    didnt see your comments until now. drop me an email. you can find it from my website linked here. http://www.es.ele.tue.nl/~sfernando/

    ReplyDelete
  18. Hi shakith,
    I am trying to run newly realeased xapp1167(http://www.xilinx.com/support/documentation/application_notes/xapp1167.pdf) for accelerating openCV with zynq on Zedboard. That app is originally for zc702 board. At first I tried to run this application as it is explained but did not work on Zedboard (expected).
    In order to run this on zed board, I created a new boot.bin file (by make boot as in app note) using the FSBL and U-Boot binary for Zedboard from xilinx wiki release Zynq 14.5. Also replaced the devicetree.dtb file in the sd_image with the one for ZedBoard from above release. Now I am able to boot the processors. but when i try to run the application i get following error
    zynq> ./run.sh
    Error : cann't open framebuffer device /dev/fb0

    The application needs to interact with hdmi port but I suspect it is somehow not detected. Can you suggest me what needs to be done in order to get the hdmi port working or simply get rid of this error.
    thanks
    umair

    ReplyDelete
  19. Maybe a drop me an email.
    http://www.es.ele.tue.nl/~sfernando/

    ReplyDelete
  20. Hi Fernando;
    I have done all of your instructions. However, I could not perform sobel filtering. After the "sudo ./camera" command,
    VIDIOC_QUERYMENU:INVALID ARGUMENT
    VIDIOC_QUERYMENU:INVALID ARGUMENT
    VIDIOC_QUERYMENU:INVALID ARGUMENT
    VIDIOC_QUERYMENU:INVALID ARGUMENT
    VIDIOC_QUERYMENU:INVALID ARGUMENT
    VIDIOC_QUERYMENU:INVALID ARGUMENT
    VIDIOC_QUERYMENU:INVALID ARGUMENT
    error occurs on terminal and also I could not realize where to use "h" or "s" for filtering options.
    Thanks in advance.

    ReplyDelete
  21. Hello Fernando,

    I am using a Zc702 Board, i want to plug a usb camera into the Zc702 board(its just a evaluation board, not a image and video tool kit from Xilinx), I have a logitech webcam c250 , I am trying to find a linux device driver for the webcam to be build with linux kernel. all i can find is prebuilt sd image. Do you have any idea how i can go about this. what can i do to plug a usb camera to zc702 board and access the frames? any inputs from you would be really helpful

    ReplyDelete
    Replies
    1. Hello,
      I have the same question of Sharath. I would like to use an USB camera witht the ZedBoard. Have you an idea about the OS to be used for that.
      Thanks

      Delete