Fix typos

Typos found with codespell.

README.org

@@ -7,25 +7,27 @@ for operations on the file.  This way multiple large files (like
 terabytes or whatever) can be instantly and simultaneously accessed
 without swapping and degraded performance.
 
-This is development version of the GNU ELPA [[http://elpa.gnu.org/packages/vlf][VLF]] package.  Here's what
+This is development version of the GNU ELPA [[https://elpa.gnu.org/packages/vlf.html][VLF]] package.  Here's what
 it offers in a nutshell:
 
-- regular expression search on whole file (in constant memory
-  determined by current batch size)
+- automatic adjustment of batch size for optimal performance and
+  responsiveness
+- regular expression search and replace over whole file
+- [[http://www.emacswiki.org/emacs/OccurMode][Occur]] like indexing
+- by batch [[http://www.emacswiki.org/emacs/EdiffMode][Ediff]] comparison
+- automatic scrolling of batches
 - chunk editing (save is immediate if size hasn't changed, done in
   constant memory determined by current batch size otherwise)
-- [[http://www.emacswiki.org/emacs/OccurMode][Occur]] like indexing
 - options to jump to beginning, end or arbitrary file chunk
+- proper dealing with multibyte encodings
+- smooth integration with [[http://www.emacswiki.org/emacs/HexlMode][hexl-mode]], just turn it on and the HEX
+  editing will work in batches just the same
+- works with [[http://www.emacswiki.org/emacs/TrampMode][TRAMP]] so accessing network files is fine and quick
 - newly added content is acknowledged if file has changed size
   meanwhile
-- automatic scrolling of batches
-- automatic adjustment of batch size for optimal performance
 - as it's a minor mode, font locking and functionality of the
-  respective major mode is also present
-- by batch [[http://www.emacswiki.org/emacs/EdiffMode][Ediff]] comparison
+  respective major mode and other minor modes is also present
 - can be added as option to automatically open large files
-- smooth integration with [[http://www.emacswiki.org/emacs/HexlMode][hexl-mode]]
-- works with [[http://www.emacswiki.org/emacs/TrampMode][TRAMP]] so accessing network files is fine
 
 GNU Emacs 23 and 24 are supported.
 
@@ -52,7 +54,7 @@ integer value).
 To have *vlf* offered as choice when opening large files:
 
 #+BEGIN_SRC emacs-lisp
-  (require 'vlf-integrate)
+  (require 'vlf-setup)
 #+END_SRC
 
 You can control when *vlf-mode* is invoked or offered with the
@@ -91,6 +93,12 @@ default.  Here's example how to add another prefix (*C-x v*):
     '(define-key vlf-prefix-map "\C-xv" vlf-mode-map))
 #+END_SRC
 
+** Overall position indicators
+
+To see which part of the file is currently visited and how many
+batches there are in overall (using the current batch size), look at
+the VLF section in the mode line, file size is also there.
+
 ** Batch size control
 
 By default *VLF* gathers statistics over how primitive operations
@@ -133,20 +141,18 @@ append prefix number of batches.
 *C-c C-v [* and *C-c C-v ]* take you to the beginning and end of file
 respectively.
 
-*C-c C-v j* jumps to given chunk.  To see where you are in file and
-how many chunks there are (using the current batch size), look at the
-parenthesized part of the buffer name, batch size is also indicated at
-the end.
+*C-c C-v j* jumps to particular batch number.
 
 ** Follow point
 
 Continuous chunk recenter around point in current buffer can be
 toggled with *C-c C-v f*.
 
-** Search whole file
+** Search and/or replace whole file
 
 *C-c C-v s* and *C-c C-v r* search forward and backward respectively
-over the whole file, batch by batch.
+over the whole file, batch by batch.  *C-c C-v %* does search and
+query replace saving intermediate changes.
 
 ** Occur over whole file
 
@@ -168,6 +174,8 @@ lines are counted from the end of file.
 
 If editing doesn't change size of the chunk, only this chunk is saved.
 Otherwise the remaining part of the file is adjusted batch by batch.
+*vlf-save-in-place* customization option controls if temporary file
+should be used in such case.
 
 ** By batch Ediff
 
@@ -178,12 +186,12 @@ last difference in current chunk searches for following one with
 difference.  The other way around if looking for difference before the
 first one.
 
-* Extending
+* Extend
 
 ** Move hooks
 
 A couple of hooks are run whenever updating chunk:
-*vlf-before-chunk-update* and *vlf-after-chunk-update*.
+*vlf-before-chunk-update-hook* and *vlf-after-chunk-update-hook*.
 
 ** Batch move hooks
 

vlf-base.el

@@ -1,6 +1,6 @@
 ;;; vlf-base.el --- VLF primitive operations  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2014 Free Software Foundation, Inc.
+;; Copyright (C) 2014-2017 Free Software Foundation, Inc.
 
 ;; Keywords: large files, chunk
 ;; Author: Andrey Kotlarski <m00naticus@gmail.com>
@@ -29,11 +29,11 @@
 
 (require 'vlf-tune)
 
-(defcustom vlf-before-chunk-update nil
+(defcustom vlf-before-chunk-update-hook nil
   "Hook that runs before chunk update."
   :group 'vlf :type 'hook)
 
-(defcustom vlf-after-chunk-update nil
+(defcustom vlf-after-chunk-update-hook nil
   "Hook that runs after chunk update."
   :group 'vlf :type 'hook)
 
@@ -47,6 +47,8 @@
 (make-variable-buffer-local 'vlf-end-pos)
 (put 'vlf-end-pos 'permanent-local t)
 
+(defvar hexl-bits)
+
 (defconst vlf-sample-size 24
   "Minimal number of bytes that can be properly decoded.")
 
@@ -54,11 +56,13 @@
   "Get size in bytes of FILE."
   (or (nth 7 (file-attributes file)) 0))
 
-(defun vlf-verify-size (&optional update-visited-time)
+(defun vlf-verify-size (&optional update-visited-time file)
   "Update file size information if necessary and visited file time.
-If non-nil, UPDATE-VISITED-TIME."
+If non-nil, UPDATE-VISITED-TIME.
+FILE if given is filename to be used, otherwise `buffer-file-truename'."
   (unless (verify-visited-file-modtime (current-buffer))
-    (setq vlf-file-size (vlf-get-file-size buffer-file-truename))
+    (setq vlf-file-size (vlf-get-file-size (or file
+                                               buffer-file-truename)))
     (if update-visited-time
         (set-visited-file-modtime))))
 
@@ -67,15 +71,6 @@ If non-nil, UPDATE-VISITED-TIME."
     "Print FILE-SIZE in MB."
     (format "%.3fMB" (/ file-size 1048576.0))))
 
-(defun vlf-update-buffer-name ()
-  "Update the current buffer name."
-  (rename-buffer (format "%s(%d/%d)[%s]"
-                         (file-name-nondirectory buffer-file-name)
-                         (/ vlf-end-pos vlf-batch-size)
-                         (/ vlf-file-size vlf-batch-size)
-                         (file-size-human-readable vlf-batch-size))
-                 t))
-
 (defmacro vlf-with-undo-disabled (&rest body)
   "Execute BODY with temporarily disabled undo."
   `(let ((undo-list buffer-undo-list))
@@ -83,32 +78,30 @@ If non-nil, UPDATE-VISITED-TIME."
      (unwind-protect (progn ,@body)
        (setq buffer-undo-list undo-list))))
 
-(defun vlf-move-to-chunk (start end &optional minimal)
+(defun vlf-move-to-chunk (start end)
   "Move to chunk enclosed by START END bytes.
-When given MINIMAL flag, skip non important operations.
 If same as current chunk is requested, do nothing.
 Return number of bytes moved back for proper decoding and number of
 bytes added to the end."
   (vlf-verify-size)
-  (cond ((or (<= end start) (<= end 0)
-             (<= vlf-file-size start))
-         (when (or (not (buffer-modified-p))
-                   (y-or-n-p "Chunk modified, are you sure? "))
-           (erase-buffer)
-           (set-buffer-modified-p nil)
-           (let ((place (if (<= vlf-file-size start)
-                            vlf-file-size
-                          0)))
-             (setq vlf-start-pos place
-                   vlf-end-pos place)
-             (or minimal (vlf-update-buffer-name))
-             (cons (- start place) (- place end)))))
-        ((or (/= start vlf-start-pos)
-             (/= end vlf-end-pos))
-         (let ((shifts (vlf-move-to-chunk-1 start end)))
-           (and shifts (not minimal)
-                (vlf-update-buffer-name))
-           shifts))))
+  (if (or (<= end start) (<= end 0)
+          (<= vlf-file-size start))
+      (when (or (not (buffer-modified-p))
+                (y-or-n-p "Chunk modified, are you sure? "))
+        (erase-buffer)
+        (set-buffer-modified-p nil)
+        (let ((place (if (<= vlf-file-size start)
+                         vlf-file-size
+                       0)))
+          (setq vlf-start-pos place
+                vlf-end-pos place)
+          (cons (- start place) (- place end))))
+    (if (derived-mode-p 'hexl-mode)
+        (setq start (- start (mod start hexl-bits))
+              end (+ end (- hexl-bits (mod end hexl-bits)))))
+    (if (or (/= start vlf-start-pos)
+            (/= end vlf-end-pos))
+        (vlf-move-to-chunk-1 start end))))
 
 (defun vlf-move-to-chunk-1 (start end)
   "Move to chunk enclosed by START END keeping as much edits if any.
@@ -133,6 +126,7 @@ bytes added to the end."
                      vlf-end-pos))
          (shifts
           (cond
+           ((and hexl (not modified)) (vlf-move-to-chunk-2 start end))
            ((or (< edit-end start) (< end vlf-start-pos)
                 (not (verify-visited-file-modtime (current-buffer))))
             (when (or (not modified)
@@ -146,10 +140,13 @@ bytes added to the end."
               (if (consp hexl-undo-list)
                   (setq hexl-undo-list nil))
               (vlf-move-to-chunk-2 start end)))
-           ((or (and (<= start vlf-start-pos) (<= edit-end end))
-                (not modified)
+           ((and (not modified)
+                 (not (consp buffer-undo-list)))
+            (vlf-move-to-chunk-2 start end))
+           ((or (not modified)
+                (and (<= start vlf-start-pos) (<= edit-end end))
                 (y-or-n-p "Chunk modified, are you sure? "))
-            (run-hooks 'vlf-before-chunk-update)
+            (run-hooks 'vlf-before-chunk-update-hook)
             (when (and hexl (not restore-hexl))
               (if (consp buffer-undo-list)
                   (setq buffer-undo-list nil))
@@ -218,7 +215,7 @@ bytes added to the end."
               (when hexl
                 (vlf-tune-hexlify)
                 (setq restore-hexl nil))
-              (run-hooks 'vlf-after-chunk-update)
+              (run-hooks 'vlf-after-chunk-update-hook)
               (cons shift-start shift-end))))))
     (when restore-hexl
       (vlf-tune-hexlify)
@@ -229,29 +226,41 @@ bytes added to the end."
   "Unconditionally move to chunk enclosed by START END bytes.
 Return number of bytes moved back for proper decoding and number of
 bytes added to the end."
-  (run-hooks 'vlf-before-chunk-update)
-  (vlf-verify-size t)
-  (setq vlf-start-pos (max 0 start)
-        vlf-end-pos (min end vlf-file-size))
-  (let (shifts)
-    (let ((inhibit-read-only t)
-          (pos (position-bytes (point))))
-      (vlf-with-undo-disabled
-       (let ((hexl (derived-mode-p 'hexl-mode)))
-         (if hexl (hexl-mode-exit t))
+  (run-hooks 'vlf-before-chunk-update-hook)
+  (let ((adjust-start t)
+        (adjust-end t)
+        (is-hexl (derived-mode-p 'hexl-mode)))
+    (and (not is-hexl)
+         (verify-visited-file-modtime (current-buffer))
+         (setq adjust-start (and (/= start vlf-start-pos)
+                                 (/= start vlf-end-pos))
+               adjust-end (and (/= end vlf-start-pos)
+                               (/= end vlf-end-pos))))
+    (vlf-verify-size t)
+    (setq vlf-start-pos (max 0 start)
+          vlf-end-pos (min end vlf-file-size))
+    (let ((shifts '(0 . 0)))
+      (let ((inhibit-read-only t)
+            (pos (position-bytes (point))))
+        (vlf-with-undo-disabled
          (erase-buffer)
-         (setq shifts (vlf-insert-file-contents vlf-start-pos
-                                                vlf-end-pos t t)
-               vlf-start-pos (- vlf-start-pos (car shifts))
-               vlf-end-pos (+ vlf-end-pos (cdr shifts)))
-         (if hexl (vlf-tune-hexlify)))
-       (goto-char (or (byte-to-position (+ pos (car shifts)))
-                      (point-max)))))
-    (set-buffer-modified-p nil)
-    (or (eq buffer-undo-list t)
-        (setq buffer-undo-list nil))
-    (run-hooks 'vlf-after-chunk-update)
-    shifts))
+         (if is-hexl
+             (progn (vlf-tune-insert-file-contents-literally
+                     vlf-start-pos vlf-end-pos)
+                    (vlf-tune-hexlify))
+           (setq shifts (vlf-insert-file-contents vlf-start-pos
+                                                  vlf-end-pos
+                                                  adjust-start
+                                                  adjust-end)
+                 vlf-start-pos (- vlf-start-pos (car shifts))
+                 vlf-end-pos (+ vlf-end-pos (cdr shifts)))))
+        (goto-char (or (byte-to-position (+ pos (car shifts)))
+                       (point-max))))
+      (set-buffer-modified-p nil)
+      (or (eq buffer-undo-list t)
+          (setq buffer-undo-list nil))
+      (run-hooks 'vlf-after-chunk-update-hook)
+      shifts)))
 
 (defun vlf-insert-file-contents (start end adjust-start adjust-end
                                        &optional position)
@@ -261,9 +270,11 @@ to the beginning, ADJUST-END - append to the end.
 Use buffer POSITION as start if given.
 Return number of bytes moved back for proper decoding and number of
 bytes added to the end."
-  (setq adjust-start (and adjust-start (not (zerop start)))
-        adjust-end (and adjust-end (< end vlf-file-size))
+  (setq adjust-end (and adjust-end (/= end vlf-file-size))
         position (or position (point-min)))
+  (and adjust-start (<= start 4)
+       (setq adjust-start nil
+             start 0))
   (goto-char position)
   (let ((shift-start 0)
         (shift-end 0)
@@ -285,7 +296,10 @@ bytes added to the end."
 
 (defun vlf-insert-file-contents-1 (start end)
   "Extract decoded file bytes START to END."
-  (vlf-tune-insert-file-contents start end))
+  (if (zerop start)
+      (vlf-tune-insert-file-contents start end)
+    (let ((coding-system-for-read buffer-file-coding-system))
+      (vlf-tune-insert-file-contents start end))))
 
 (defun vlf-adjust-start (start end position adjust-end)
   "Adjust chunk beginning at absolute START to END till content can\
@@ -298,9 +312,10 @@ Return number of bytes moved back for proper decoding."
          (strict (or (= sample-end vlf-file-size)
                      (and (not adjust-end) (= sample-end end))))
          (shift 0))
-    (while (and (progn (vlf-insert-file-contents-1 safe-start
-                                                   sample-end)
-                       (not (zerop safe-start)))
+    (while (and (let ((coding-system-for-read buffer-file-coding-system))
+                  (insert-file-contents buffer-file-name
+                                        nil safe-start sample-end)
+                  (not (zerop safe-start)))
                 (< shift 3)
                 (let ((diff (- chunk-size
                                (length
@@ -366,6 +381,14 @@ which deletion was performed."
        (delete-region cut-point (point-max))))
     (cons dist (1+ cut-point))))
 
+(defun vlf-byte-position (point)
+  "Determine global byte position of POINT."
+  (let ((pmax (point-max)))
+    (if (< (/ pmax 2) point)
+        (- vlf-end-pos (vlf-tune-encode-length (min (1+ point) pmax)
+                                               pmax))
+      (+ vlf-start-pos (vlf-tune-encode-length (point-min) point)))))
+
 (defun vlf-shift-undo-list (n)
   "Shift undo list element regions by N."
   (or (null buffer-undo-list) (eq buffer-undo-list t)

vlf-ediff.el

@@ -34,6 +34,8 @@
   "If non nil, specifies that ediff is done over VLF buffers.")
 (make-variable-buffer-local 'vlf-ediff-session)
 
+(defvar tramp-verbose)
+
 (defun vlf-ediff-buffers (buffer-A buffer-B)
   "Run batch by batch ediff over VLF buffers BUFFER-A and BUFFER-B.
 Batch size is determined by the size in BUFFER-A.
@@ -141,11 +143,11 @@ beginning of difference list."
 
 (defun vlf-next-chunk ()
   "Move to next chunk."
-  (vlf-move-to-chunk vlf-end-pos (+ vlf-end-pos vlf-batch-size) t))
+  (vlf-move-to-chunk vlf-end-pos (+ vlf-end-pos vlf-batch-size)))
 
 (defun vlf-prev-chunk ()
   "Move to previous chunk."
-  (vlf-move-to-chunk (- vlf-start-pos vlf-batch-size) vlf-start-pos t))
+  (vlf-move-to-chunk (- vlf-start-pos vlf-batch-size) vlf-start-pos))
 
 (defun vlf-ediff-next (buffer-A buffer-B ediff-buffer
                                 &optional next-func)
@@ -168,8 +170,9 @@ logical chunks in case there is no difference at the current ones."
     (setq buffer-B (current-buffer)
           min-file-size (min min-file-size vlf-file-size)
           is-hexl (or is-hexl (derived-mode-p 'hexl-mode)))
-    (let ((tramp-verbose (if (boundp 'tramp-verbose)
-                             (min tramp-verbose 2)))
+    (let ((tramp-verbose (if (and (boundp 'tramp-verbose)
+                                  tramp-verbose)
+                             (min tramp-verbose 1)))
           (end-B (= vlf-start-pos vlf-end-pos))
           (chunk-B (cons vlf-start-pos vlf-end-pos))
           (font-lock-B font-lock-mode)
@@ -203,10 +206,7 @@ logical chunks in case there is no difference at the current ones."
                                           (- vlf-file-size
                                              vlf-start-pos))))
             (progress-reporter-done reporter)
-            (if (or (not end-A) (not end-B))
-                (progn (vlf-update-buffer-name)
-                       (set-buffer buffer-A)
-                       (vlf-update-buffer-name))
+            (when (and end-A end-B)
               (if forward-p
                   (let ((max-file-size vlf-file-size))
                     (vlf-move-to-chunk (- max-file-size vlf-batch-size)
@@ -217,9 +217,9 @@ logical chunks in case there is no difference at the current ones."
                     (vlf-move-to-chunk (- max-file-size
                                           vlf-batch-size)
                                        max-file-size))
-                (vlf-beginning-of-file)
+                (vlf-move-to-batch 0)
                 (set-buffer buffer-A)
-                (vlf-beginning-of-file))
+                (vlf-move-to-batch 0))
               (set-buffer ediff-buffer)
               (ediff-update-diffs)
               (or is-hexl

vlf-occur.el

@@ -44,6 +44,9 @@
 (defvar vlf-occur-lines 0 "Number of lines scanned by `vlf-occur'.")
 (make-variable-buffer-local 'vlf-occur-lines)
 
+(defvar tramp-verbose)
+(defvar hexl-bits)
+
 (defvar vlf-occur-mode-map
   (let ((map (make-sparse-keymap)))
     (define-key map "n" 'vlf-occur-next-match)
@@ -107,7 +110,8 @@ EVENT may hold details of the invocation."
     (set-buffer (window-buffer (posn-window (event-end event))))
     (goto-char (posn-point (event-end event))))
   (let* ((pos (point))
-         (pos-relative (- pos (line-beginning-position) 1))
+         (pos-relative (- pos (previous-single-char-property-change
+                               pos 'vlf-match)))
          (chunk-start (get-text-property pos 'chunk-start)))
     (if chunk-start
         (let ((chunk-end (get-text-property pos 'chunk-end))
@@ -123,8 +127,11 @@ EVENT may hold details of the invocation."
               (match-pos (+ (get-text-property pos 'line-pos)
                             pos-relative)))
           (cond (current-prefix-arg
-                 (setq vlf-buffer (vlf file t))
-                 (or not-hexl (vlf-tune-hexlify))
+                 (let ((original-occur-buffer vlf-occur-vlf-buffer))
+                   (setq vlf-buffer (vlf file t))
+                   (if (buffer-live-p original-occur-buffer)
+                       (vlf-tune-copy-profile original-occur-buffer)))
+                 (or not-hexl (hexl-mode))
                  (switch-to-buffer occur-buffer))
                 ((not (buffer-live-p vlf-buffer))
                  (unless (catch 'found
@@ -137,7 +144,7 @@ EVENT may hold details of the invocation."
                                   (setq vlf-buffer buf)
                                   (throw 'found t))))
                    (setq vlf-buffer (vlf file t))
-                   (or not-hexl (vlf-tune-hexlify)))
+                   (or not-hexl (hexl-mode)))
                  (switch-to-buffer occur-buffer)
                  (setq vlf-occur-vlf-buffer vlf-buffer)))
           (pop-to-buffer vlf-buffer)
@@ -149,43 +156,27 @@ EVENT may hold details of the invocation."
 Prematurely ending indexing will still show what's found so far."
   (let ((vlf-buffer (current-buffer))
         (file buffer-file-name)
+        (file-size vlf-file-size)
         (batch-size vlf-batch-size)
-        (is-hexl (derived-mode-p 'hexl-mode))
-        (insert-bps vlf-tune-insert-bps)
-        (encode-bps vlf-tune-encode-bps)
-        (hexl-bps vlf-tune-hexl-bps)
-        (dehexlify-bps vlf-tune-dehexlify-bps))
+        (is-hexl (derived-mode-p 'hexl-mode)))
     (with-temp-buffer
       (setq buffer-file-name file
             buffer-file-truename file
-            buffer-undo-list t)
+            buffer-undo-list t
+            vlf-file-size file-size)
       (set-buffer-modified-p nil)
       (set (make-local-variable 'vlf-batch-size) batch-size)
       (when vlf-tune-enabled
-        (setq vlf-tune-insert-bps insert-bps
-              vlf-tune-encode-bps encode-bps)
-        (if is-hexl
-            (progn (setq vlf-tune-hexl-bps hexl-bps
-                         vlf-tune-dehexlify-bps dehexlify-bps)
-                   (vlf-tune-batch '(:hexl :dehexlify :insert :encode)))
-          (vlf-tune-batch '(:insert :encode))))
+        (vlf-tune-copy-profile vlf-buffer)
+        (vlf-tune-batch (if is-hexl
+                            '(:hexl :raw)
+                          '(:insert :encode)) t))
       (vlf-mode 1)
-      (if is-hexl (vlf-tune-hexlify))
+      (if is-hexl (hexl-mode))
       (goto-char (point-min))
-      (vlf-with-undo-disabled
-       (vlf-build-occur regexp vlf-buffer))
-      (when vlf-tune-enabled
-        (setq insert-bps vlf-tune-insert-bps
-              encode-bps vlf-tune-encode-bps)
-        (if is-hexl
-            (setq insert-bps vlf-tune-insert-bps
-                  encode-bps vlf-tune-encode-bps))))
-    (when vlf-tune-enabled              ;merge back tune measurements
-      (setq vlf-tune-insert-bps insert-bps
-            vlf-tune-encode-bps encode-bps)
-      (if is-hexl
-          (setq vlf-tune-insert-bps insert-bps
-                vlf-tune-encode-bps encode-bps)))))
+      (vlf-build-occur regexp vlf-buffer)
+      (if vlf-tune-enabled
+          (vlf-tune-copy-profile (current-buffer) vlf-buffer)))))
 
 (defun vlf-occur (regexp)
   "Make whole file occur style index for REGEXP.
@@ -195,47 +186,46 @@ Prematurely ending indexing will still show what's found so far."
                                       (car regexp-history)))))
   (run-hook-with-args 'vlf-before-batch-functions 'occur)
   (if (or (buffer-modified-p)
+          (consp buffer-undo-list)
           (< vlf-batch-size vlf-start-pos))
       (vlf-occur-other-buffer regexp)
     (let ((start-pos vlf-start-pos)
           (end-pos vlf-end-pos)
           (pos (point))
-          (batch-size vlf-batch-size)
-          (is-hexl (derived-mode-p 'hexl-mode)))
+          (batch-size vlf-batch-size))
       (vlf-tune-batch (if (derived-mode-p 'hexl-mode)
-                          '(:hexl :dehexlify :insert :encode)
-                        '(:insert :encode)))
-      (vlf-with-undo-disabled
-       (vlf-move-to-batch 0)
-       (goto-char (point-min))
-       (unwind-protect (vlf-build-occur regexp (current-buffer))
-         (vlf-move-to-chunk start-pos end-pos)
-         (if is-hexl (vlf-tune-hexlify))
-         (goto-char pos)
-         (setq vlf-batch-size batch-size)))))
+                          '(:hexl :raw)
+                        '(:insert :encode)) t)
+      (vlf-move-to-batch 0)
+      (goto-char (point-min))
+      (unwind-protect (vlf-build-occur regexp (current-buffer))
+        (vlf-move-to-chunk start-pos end-pos)
+        (goto-char pos)
+        (setq vlf-batch-size batch-size))))
   (run-hook-with-args 'vlf-after-batch-functions 'occur))
 
 (defun vlf-build-occur (regexp vlf-buffer)
   "Build occur style index for REGEXP over VLF-BUFFER."
-  (let* ((tramp-verbose (if (boundp 'tramp-verbose)
-                            (min tramp-verbose 2)))
+  (let* ((tramp-verbose (if (and (boundp 'tramp-verbose)
+                                 tramp-verbose)
+                            (min tramp-verbose 1)))
          (case-fold-search t)
          (line 1)
          (last-match-line 0)
-         (last-line-pos (point-min))
          (total-matches 0)
-         (match-end-pos (+ vlf-start-pos (position-bytes (point))))
+         (first-line-offset 0)
+         (first-line-incomplete nil)
+         (match-start-point (point-min))
+         (match-end-point match-start-point)
+         (last-match-insert-point nil)
          (occur-buffer (generate-new-buffer
                         (concat "*VLF-occur " (file-name-nondirectory
                                                buffer-file-name)
                                 "*")))
-         (line-regexp (concat "\\(?5:[\n\C-m]\\)\\|\\(?10:"
-                              regexp "\\)"))
-         (batch-step (min 1024 (/ vlf-batch-size 8)))
          (is-hexl (derived-mode-p 'hexl-mode))
          (end-of-file nil)
          (time (float-time))
-         (tune-types (if is-hexl '(:hexl :dehexlify :insert :encode)
+         (tune-types (if is-hexl '(:hexl :raw)
                        '(:insert :encode)))
          (reporter (make-progress-reporter
                     (concat "Building index for " regexp "...")
@@ -245,69 +235,117 @@ Prematurely ending indexing will still show what's found so far."
     (unwind-protect
         (progn
           (while (not end-of-file)
-            (if (re-search-forward line-regexp nil t)
+            (if (re-search-forward regexp nil t)
                 (progn
-                  (setq match-end-pos (+ vlf-start-pos
-                                         (position-bytes
-                                          (match-end 0))))
-                  (if (match-string 5)
-                      (setq line (1+ line) ; line detected
-                            last-line-pos (point))
-                    (let* ((chunk-start vlf-start-pos)
-                           (chunk-end vlf-end-pos)
-                           (line-pos (line-beginning-position))
-                           (line-text (buffer-substring
-                                       line-pos (line-end-position))))
-                      (with-current-buffer occur-buffer
-                        (unless (= line last-match-line) ;new match line
-                          (insert "\n:") ; insert line number
-                          (let* ((overlay-pos (1- (point)))
-                                 (overlay (make-overlay
-                                           overlay-pos
-                                           (1+ overlay-pos))))
-                            (overlay-put overlay 'before-string
-                                         (propertize
-                                          (number-to-string line)
-                                          'face 'shadow)))
-                          (insert (propertize line-text ; insert line
-                                              'chunk-start chunk-start
-                                              'chunk-end chunk-end
-                                              'mouse-face '(highlight)
-                                              'line-pos line-pos
-                                              'help-echo
-                                              (format "Move to line %d"
-                                                      line))))
-                        (setq last-match-line line
-                              total-matches (1+ total-matches))
-                        (let ((line-start (1+
-                                           (line-beginning-position)))
-                              (match-pos (match-beginning 10)))
-                          (add-text-properties ; mark match
-                           (+ line-start match-pos (- last-line-pos))
-                           (+ line-start (match-end 10)
-                              (- last-line-pos))
-                           (list 'face 'match
+                  (setq line (+ line -1
+                                (count-lines match-start-point
+                                             (1+ (match-beginning 0))))
+                        match-start-point (match-beginning 0)
+                        match-end-point (match-end 0))
+                  (let* ((chunk-start vlf-start-pos)
+                         (chunk-end vlf-end-pos)
+                         (line-pos (save-excursion
+                                     (goto-char match-start-point)
+                                     (line-beginning-position)))
+                         (line-text (buffer-substring
+                                     line-pos (line-end-position))))
+                    (if (/= line-pos (point-min))
+                        (setq first-line-offset 0
+                              first-line-incomplete nil))
+                    (with-current-buffer occur-buffer
+                      (unless (= line last-match-line) ;new match line
+                        (insert "\n:") ; insert line number
+                        (let* ((column-point (1- (point)))
+                               (overlay-pos column-point)
+                               (overlay (make-overlay
+                                         overlay-pos
+                                         (1+ overlay-pos))))
+                          (overlay-put overlay 'before-string
+                                       (propertize
+                                        (number-to-string line)
+                                        'face 'shadow))
+                          (overlay-put overlay 'vlf-match t)
+                          (setq last-match-insert-point column-point
+                                first-line-offset 0)))
+                      (when (or first-line-incomplete
+                                (/= line last-match-line))
+                        (insert (propertize
+                                 (if first-line-incomplete
+                                     (substring line-text
+                                                first-line-incomplete)
+                                   line-text)
+                                 'chunk-start chunk-start
+                                 'chunk-end chunk-end
+                                 'mouse-face '(highlight)
+                                 'line-pos line-pos
                                  'help-echo
-                                 (format "Move to match %d"
-                                         total-matches))))))))
+                                 (format "Move to line %d"
+                                         line)))
+                        (setq first-line-incomplete nil))
+                      (setq last-match-line line
+                            total-matches (1+ total-matches))
+                      (let ((line-start (+ last-match-insert-point
+                                           first-line-offset 1
+                                           (- line-pos))))
+                        (add-text-properties ; mark match
+                         (+ line-start match-start-point)
+                         (+ line-start match-end-point)
+                         (list 'face 'match
+                               'help-echo (format "Move to match %d"
+                                                  total-matches)))))))
               (setq end-of-file (= vlf-end-pos vlf-file-size))
               (unless end-of-file
-                (vlf-tune-batch tune-types)
-                (let ((batch-move (- vlf-end-pos batch-step)))
-                  (vlf-move-to-batch (if (or is-hexl
-                                             (< match-end-pos
-                                                batch-move))
-                                         batch-move
-                                       match-end-pos) t))
-                (goto-char (if (or is-hexl
-                                   (<= match-end-pos vlf-start-pos))
-                               (point-min)
-                             (or (byte-to-position (- match-end-pos
-                                                      vlf-start-pos))
-                                 (point-min))))
-                (setq last-match-line 0
-                      last-line-pos (line-beginning-position))
-                (progress-reporter-update reporter vlf-end-pos))))
+                (let ((start
+                       (if is-hexl
+                           (progn
+                             (goto-char (point-max))
+                             (forward-line -10)
+                             (setq line
+                                   (+ line
+                                      (if (< match-end-point (point))
+                                          (count-lines match-start-point
+                                                       (point))
+                                        (goto-char match-end-point)
+                                        (1- (count-lines match-start-point
+                                                         match-end-point)))))
+                             (- vlf-end-pos (* (- 10 (forward-line 10))
+                                               hexl-bits)))
+                         (let* ((pmax (point-max))
+                                (batch-step (min 1024 (/ vlf-batch-size
+                                                         10)))
+                                (batch-point
+                                 (max match-end-point
+                                      (or
+                                       (byte-to-position
+                                        (- vlf-batch-size batch-step))
+                                       (progn
+                                         (goto-char pmax)
+                                         (let ((last (line-beginning-position)))
+                                           (if (= last (point-min))
+                                               (1- (point))
+                                             last)))))))
+                           (goto-char batch-point)
+                           (setq first-line-offset
+                                 (- batch-point (line-beginning-position))
+                                 line
+                                 (+ line
+                                    (count-lines match-start-point
+                                                 batch-point)
+                                    (if (< 0 first-line-offset) -1 0)))
+                           ;; last match is on the last line?
+                           (goto-char match-end-point)
+                           (forward-line)
+                           (setq first-line-incomplete
+                                 (if (= (point) pmax)
+                                     (- pmax match-end-point)))
+                           (vlf-byte-position batch-point)))))
+                  (vlf-tune-batch tune-types)
+                  (setq vlf-end-pos start) ;not to adjust start
+                  (vlf-move-to-chunk start (+ start vlf-batch-size)))
+                (setq match-start-point (point-min)
+                      match-end-point match-start-point)
+                (goto-char match-end-point)
+                (progress-reporter-update reporter vlf-start-pos))))
           (progress-reporter-done reporter))
       (set-buffer-modified-p nil)
       (if (zerop total-matches)
@@ -317,6 +355,7 @@ Prematurely ending indexing will still show what's found so far."
         (let ((file buffer-file-name)
               (dir default-directory))
           (with-current-buffer occur-buffer
+            (insert "\n")
             (goto-char (point-min))
             (insert (propertize
                      (format "%d matches from %d lines for \"%s\" \
@@ -361,17 +400,20 @@ in file: %s" total-matches line regexp file)
            vlf-occur-save-buffer)
     (save-excursion
       (goto-char (point-min))
-      (while (zerop (forward-line))
-        (let* ((pos (1+ (point)))
-               (line (get-char-property (1- pos) 'before-string)))
-          (if line
-              (prin1 (list (string-to-number line)
-                           (get-text-property pos 'chunk-start)
-                           (get-text-property pos 'chunk-end)
-                           (get-text-property pos 'line-pos)
-                           (buffer-substring-no-properties
-                            pos (line-end-position)))
-                     vlf-occur-save-buffer)))))
+      (let ((pmax (point-max)))
+        (while (/= pmax (goto-char (next-single-char-property-change
+                                    (1+ (point)) 'vlf-match)))
+          (let* ((pos (1+ (point)))
+                 (line (get-char-property (1- pos) 'before-string)))
+            (if line
+                (prin1 (list (string-to-number line)
+                             (get-text-property pos 'chunk-start)
+                             (get-text-property pos 'chunk-end)
+                             (get-text-property pos 'line-pos)
+                             (buffer-substring-no-properties
+                              pos (1- (next-single-char-property-change
+                                       pos 'vlf-match))))
+                       vlf-occur-save-buffer))))))
     (with-current-buffer vlf-occur-save-buffer
       (save-buffer))
     (kill-buffer vlf-occur-save-buffer))
@@ -407,6 +449,7 @@ in file: %s" total-matches line regexp file)
                (pos (point)))
           (overlay-put overlay 'before-string
                        (propertize line 'face 'shadow))
+          (overlay-put overlay 'vlf-match t)
           (insert (propertize (nth 4 form) 'chunk-start (nth 1 form)
                               'chunk-end (nth 2 form)
                               'mouse-face '(highlight)

vlf-search.el

@@ -29,11 +29,14 @@
 
 (require 'vlf)
 
-(defun vlf-re-search (regexp count backward batch-step
-                             &optional reporter time)
+(defvar hexl-bits)
+(defvar tramp-verbose)
+
+(defun vlf-re-search (regexp count backward
+                             &optional reporter time highlight)
   "Search for REGEXP COUNT number of times forward or BACKWARD.
-BATCH-STEP is amount of overlap between successive chunks.
 Use existing REPORTER and start TIME if given.
+Highlight match if HIGHLIGHT is non nil.
 Return t if search has been at least partially successful."
   (if (<= count 0)
       (error "Count must be positive"))
@@ -45,16 +48,18 @@ Return t if search has been at least partially successful."
                                  vlf-start-pos)
                                vlf-file-size)))
   (or time (setq time (float-time)))
-  (let* ((tramp-verbose (if (boundp 'tramp-verbose)
-                            (min tramp-verbose 2)))
+  (let* ((tramp-verbose (if (and (boundp 'tramp-verbose)
+                                 tramp-verbose)
+                            (min tramp-verbose 1)))
          (case-fold-search t)
          (match-chunk-start vlf-start-pos)
          (match-chunk-end vlf-end-pos)
-         (match-start-pos (+ vlf-start-pos (position-bytes (point))))
+         (match-start-pos (point))
          (match-end-pos match-start-pos)
+         (last-match-pos match-start-pos)
          (to-find count)
          (is-hexl (derived-mode-p 'hexl-mode))
-         (tune-types (if is-hexl '(:hexl :dehexlify :insert :encode)
+         (tune-types (if is-hexl '(:hexl :raw)
                        '(:insert :encode)))
          (font-lock font-lock-mode))
     (font-lock-mode 0)
@@ -67,31 +72,31 @@ Return t if search has been at least partially successful."
                         (setq to-find (1- to-find)
                               match-chunk-start vlf-start-pos
                               match-chunk-end vlf-end-pos
-                              match-start-pos (+ vlf-start-pos
-                                                 (position-bytes
-                                                  (match-beginning 0)))
-                              match-end-pos (+ vlf-start-pos
-                                               (position-bytes
-                                                (match-end 0)))))
+                              match-start-pos (match-beginning 0)
+                              match-end-pos (match-end 0)
+                              last-match-pos match-start-pos))
                        ((zerop vlf-start-pos)
                         (throw 'end-of-file nil))
-                       (t (vlf-tune-batch tune-types)
-                          (let ((batch-move (- vlf-start-pos
-                                               (- vlf-batch-size
-                                                  batch-step))))
-                            (vlf-move-to-batch
-                             (if (or is-hexl
-                                     (<= batch-move match-start-pos))
-                                 batch-move
-                               (- match-start-pos vlf-batch-size)) t))
-                          (goto-char (if (or is-hexl
-                                             (<= vlf-end-pos
-                                                 match-start-pos))
-                                         (point-max)
-                                       (or (byte-to-position
-                                            (- match-start-pos
-                                               vlf-start-pos))
-                                           (point-max))))
+                       (t (let ((end
+                                 (if is-hexl
+                                     (progn
+                                       (goto-char (point-min))
+                                       (forward-line 10)
+                                       (if (< last-match-pos (point))
+                                           (goto-char last-match-pos))
+                                       (+ vlf-start-pos
+                                          (* (- 10 (forward-line -10))
+                                             hexl-bits)))
+                                   (vlf-byte-position
+                                    (min 1024 (/ (point-max) 10)
+                                         last-match-pos)))))
+                            (vlf-tune-batch tune-types)
+                            (setq vlf-start-pos end) ;don't adjust end
+                            (vlf-move-to-chunk (- end vlf-batch-size)
+                                               end))
+                          (let ((pmax (point-max)))
+                            (goto-char pmax)
+                            (setq last-match-pos pmax))
                           (progress-reporter-update
                            reporter (- vlf-file-size
                                        vlf-start-pos)))))
@@ -100,84 +105,80 @@ Return t if search has been at least partially successful."
                       (setq to-find (1- to-find)
                             match-chunk-start vlf-start-pos
                             match-chunk-end vlf-end-pos
-                            match-start-pos (+ vlf-start-pos
-                                               (position-bytes
-                                                (match-beginning 0)))
-                            match-end-pos (+ vlf-start-pos
-                                             (position-bytes
-                                              (match-end 0)))))
-                     ((= vlf-end-pos vlf-file-size)
+                            match-start-pos (match-beginning 0)
+                            match-end-pos (match-end 0)
+                            last-match-pos match-end-pos))
+                     ((>= vlf-end-pos vlf-file-size)
                       (throw 'end-of-file nil))
-                     (t (vlf-tune-batch tune-types)
-                        (let ((batch-move (- vlf-end-pos batch-step)))
-                          (vlf-move-to-batch
-                           (if (or is-hexl
-                                   (< match-end-pos batch-move))
-                               batch-move
-                             match-end-pos) t))
-                        (goto-char (if (or is-hexl
-                                           (<= match-end-pos vlf-start-pos))
-                                       (point-min)
-                                     (or (byte-to-position
-                                          (- match-end-pos
-                                             vlf-start-pos))
-                                         (point-min))))
+                     (t (let* ((pmax (point-max))
+                               (start
+                                (if is-hexl
+                                    (progn
+                                      (goto-char pmax)
+                                      (forward-line -10)
+                                      (if (< (point) last-match-pos)
+                                          (goto-char last-match-pos))
+                                      (- vlf-end-pos
+                                         (* (- 10 (forward-line 10))
+                                            hexl-bits)))
+                                  (vlf-byte-position
+                                   (max (- pmax 1024)
+                                        (- pmax (/ pmax 10))
+                                        last-match-pos)))))
+                          (vlf-tune-batch tune-types)
+                          (setq vlf-end-pos start) ;don't adjust start
+                          (vlf-move-to-chunk start (+ start
+                                                      vlf-batch-size)))
+                        (let ((pmin (point-min)))
+                          (goto-char pmin)
+                          (setq last-match-pos pmin))
                         (progress-reporter-update reporter
                                                   vlf-end-pos)))))
            (progress-reporter-done reporter))
        (set-buffer-modified-p nil)
-       (if is-hexl (vlf-tune-hexlify))
        (if font-lock (font-lock-mode 1))
        (let ((result
               (if backward
                   (vlf-goto-match match-chunk-start match-chunk-end
-                                  match-end-pos match-start-pos
-                                  count to-find time)
+                                  match-start-pos match-end-pos
+                                  count to-find time highlight)
                 (vlf-goto-match match-chunk-start match-chunk-end
-                                match-start-pos match-end-pos
-                                count to-find time))))
+                                match-end-pos match-start-pos
+                                count to-find time highlight))))
          (run-hook-with-args 'vlf-after-batch-functions 'search)
          result)))))
 
 (defun vlf-goto-match (match-chunk-start match-chunk-end
-                                         match-pos-start match-pos-end
-                                         count to-find time)
+                                         match-start-pos match-end-pos
+                                         count to-find time
+                                         highlight)
   "Move to MATCH-CHUNK-START MATCH-CHUNK-END surrounding\
-MATCH-POS-START and MATCH-POS-END.
+MATCH-START-POS and MATCH-END-POS.
 According to COUNT and left TO-FIND, show if search has been
 successful.  Use start TIME to report how much it took.
+Highlight match if HIGHLIGHT is non nil.
 Return nil if nothing found."
+  (vlf-move-to-chunk match-chunk-start match-chunk-end)
+  (goto-char match-start-pos)
+  (setq vlf-batch-size (vlf-tune-optimal-load
+                        (if (derived-mode-p 'hexl-mode)
+                            '(:hexl :raw)
+                          '(:insert :encode))))
   (if (= count to-find)
-      (progn (vlf-move-to-chunk match-chunk-start match-chunk-end)
-             (goto-char (or (byte-to-position (- match-pos-start
-                                                 vlf-start-pos))
-                            (point-max)))
-             (message "Not found (%f secs)" (- (float-time) time))
+      (progn (message "Not found (%f secs)" (- (float-time) time))
              nil)
-    (let ((success (zerop to-find)))
+    (let ((success (zerop to-find))
+          (overlay (make-overlay match-start-pos match-end-pos)))
+      (overlay-put overlay 'face 'match)
       (if success
-          (vlf-update-buffer-name)
-        (vlf-move-to-chunk match-chunk-start match-chunk-end))
-      (setq vlf-batch-size (vlf-tune-optimal-load
-                            (if (derived-mode-p 'hexl-mode)
-                                '(:hexl :dehexlify :insert :encode)
-                              '(:insert :encode))))
-      (let* ((match-end (or (byte-to-position (- match-pos-end
-                                                 vlf-start-pos))
-                            (point-max)))
-             (overlay (make-overlay (byte-to-position
-                                     (- match-pos-start
-                                        vlf-start-pos))
-                                    match-end)))
-        (overlay-put overlay 'face 'match)
-        (if success
-            (message "Match found (%f secs)" (- (float-time) time))
-          (goto-char match-end)
-          (message "Moved to the %d match which is last (%f secs)"
-                   (- count to-find) (- (float-time) time)))
-        (unwind-protect (sit-for 3)
-          (delete-overlay overlay))
-        t))))
+          (message "Match found (%f secs)" (- (float-time) time))
+        (message "Moved to the %d match which is last (%f secs)"
+                 (- count to-find) (- (float-time) time)))
+      (if highlight
+          (unwind-protect (sit-for 1)
+            (delete-overlay overlay))
+        (delete-overlay overlay)))
+    t))
 
 (defun vlf-re-search-forward (regexp count)
   "Search forward for REGEXP prefix COUNT number of times.
@@ -187,9 +188,11 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                                       (if regexp-history
                                           (car regexp-history)))
                          (or current-prefix-arg 1))))
-  (let ((batch-size vlf-batch-size))
-    (or (vlf-re-search regexp count nil (min 1024 (/ vlf-batch-size 8)))
-        (setq vlf-batch-size batch-size))))
+  (let ((batch-size vlf-batch-size)
+        success)
+    (unwind-protect
+        (setq success (vlf-re-search regexp count nil nil nil t))
+      (or success (setq vlf-batch-size batch-size)))))
 
 (defun vlf-re-search-backward (regexp count)
   "Search backward for REGEXP prefix COUNT number of times.
@@ -199,99 +202,153 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                                       (if regexp-history
                                           (car regexp-history)))
                          (or current-prefix-arg 1))))
-  (let ((batch-size vlf-batch-size))
-    (or (vlf-re-search regexp count t (min 1024 (/ vlf-batch-size 8)))
-        (setq vlf-batch-size batch-size))))
+  (let ((batch-size vlf-batch-size)
+        success)
+    (unwind-protect
+        (setq success (vlf-re-search regexp count t nil nil t))
+      (or success (setq vlf-batch-size batch-size)))))
 
 (defun vlf-goto-line (n)
   "Go to line N.  If N is negative, count from the end of file."
   (interactive (if (vlf-no-modifications)
                    (list (read-number "Go to line: "))))
-  (run-hook-with-args 'vlf-before-batch-functions 'goto-line)
-  (vlf-verify-size)
-  (let ((tramp-verbose (if (boundp 'tramp-verbose)
-                           (min tramp-verbose 2)))
-        (start-pos vlf-start-pos)
-        (end-pos vlf-end-pos)
-        (batch-size vlf-batch-size)
-        (pos (point))
-        (is-hexl (derived-mode-p 'hexl-mode))
-        (font-lock font-lock-mode)
-        (time (float-time))
-        (success nil))
-    (font-lock-mode 0)
-    (vlf-tune-batch '(:raw))
-    (unwind-protect
-        (if (< 0 n)
-            (let ((start 0)
-                  (end (min vlf-batch-size vlf-file-size))
+  (if (derived-mode-p 'hexl-mode)
+      (vlf-goto-line-hexl n)
+    (run-hook-with-args 'vlf-before-batch-functions 'goto-line)
+    (vlf-verify-size)
+    (let ((tramp-verbose (if (and (boundp 'tramp-verbose)
+                                  tramp-verbose)
+                             (min tramp-verbose 1)))
+          (start-pos vlf-start-pos)
+          (end-pos vlf-end-pos)
+          (batch-size vlf-batch-size)
+          (pos (point))
+          (font-lock font-lock-mode)
+          (time (float-time))
+          (success nil))
+      (font-lock-mode 0)
+      (vlf-tune-batch '(:raw))
+      (unwind-protect
+          (if (< 0 n)
+              (let ((start 0)
+                    (end (min vlf-batch-size vlf-file-size))
+                    (reporter (make-progress-reporter
+                               (concat "Searching for line "
+                                       (number-to-string n) "...")
+                               0 vlf-file-size))
+                    (inhibit-read-only t))
+                (setq n (1- n))
+                (vlf-with-undo-disabled
+                 ;; (while (and (< (- end start) n)
+                 ;;             (< n (- vlf-file-size start)))
+                 ;;   (erase-buffer)
+                 ;;   (vlf-tune-insert-file-contents-literally start end)
+                 ;;   (goto-char (point-min))
+                 ;;   (while (re-search-forward "[\n\C-m]" nil t)
+                 ;;     (setq n (1- n)))
+                 ;;   (vlf-verify-size)
+                 ;;   (vlf-tune-batch '(:raw))
+                 ;;   (setq start end
+                 ;;         end (min vlf-file-size (+ start
+                 ;;                                   vlf-batch-size)))
+                 ;;   (progress-reporter-update reporter start))
+                 (when (< n (- vlf-file-size end))
+                   (vlf-tune-batch '(:insert :encode))
+                   (vlf-move-to-chunk start (+ start vlf-batch-size))
+                   (goto-char (point-min))
+                   (setq success
+                         (or (zerop n)
+                             (when (vlf-re-search "[\n\C-m]" n nil
+                                                  reporter time)
+                               (forward-char) t))))))
+            (let ((end vlf-file-size)
                   (reporter (make-progress-reporter
-                             (concat "Searching for line "
+                             (concat "Searching for line -"
                                      (number-to-string n) "...")
                              0 vlf-file-size))
                   (inhibit-read-only t))
-              (setq n (1- n))
+              (setq n (- n))
               (vlf-with-undo-disabled
-               (or is-hexl
-                   (while (and (< (- end start) n)
-                               (< n (- vlf-file-size start)))
-                     (erase-buffer)
-                     (vlf-tune-insert-file-contents-literally start end)
-                     (goto-char (point-min))
-                     (while (re-search-forward "[\n\C-m]" nil t)
-                       (setq n (1- n)))
-                     (vlf-verify-size)
-                     (vlf-tune-batch '(:raw))
-                     (setq start end
-                           end (min vlf-file-size
-                                    (+ start vlf-batch-size)))
-                     (progress-reporter-update reporter start)))
-               (when (< n (- vlf-file-size end))
-                 (vlf-tune-batch (if is-hexl
-                                     '(:hexl :dehexlify :insert :encode)
-                                   '(:insert :encode)))
-                 (vlf-move-to-chunk-2 start (+ start vlf-batch-size))
-                 (goto-char (point-min))
-                 (setq success (vlf-re-search "[\n\C-m]" n nil 0
-                                              reporter time)))))
-          (let ((start (max 0 (- vlf-file-size vlf-batch-size)))
-                (end vlf-file-size)
-                (reporter (make-progress-reporter
-                           (concat "Searching for line -"
-                                   (number-to-string n) "...")
-                           0 vlf-file-size))
-                (inhibit-read-only t))
-            (setq n (- n))
-            (vlf-with-undo-disabled
-             (or is-hexl
-                 (while (and (< (- end start) n) (< n end))
-                   (erase-buffer)
-                   (vlf-tune-insert-file-contents-literally start end)
-                   (goto-char (point-max))
-                   (while (re-search-backward "[\n\C-m]" nil t)
-                     (setq n (1- n)))
-                   (vlf-tune-batch '(:raw))
-                   (setq end start
-                         start (max 0 (- end vlf-batch-size)))
-                   (progress-reporter-update reporter
-                                             (- vlf-file-size end))))
-             (when (< n end)
-               (vlf-tune-batch (if is-hexl
-                                   '(:hexl :dehexlify :insert :encode)
-                                 '(:insert :encode)))
-               (vlf-move-to-chunk-2 (- end vlf-batch-size) end)
-               (goto-char (point-max))
-               (setq success (vlf-re-search "[\n\C-m]" n t 0
-                                            reporter time))))))
-      (if font-lock (font-lock-mode 1))
-      (unless success
-        (vlf-with-undo-disabled
-         (vlf-move-to-chunk-2 start-pos end-pos))
-        (vlf-update-buffer-name)
-        (goto-char pos)
-        (setq vlf-batch-size batch-size)
-        (message "Unable to find line"))
-      (run-hook-with-args 'vlf-after-batch-functions 'goto-line))))
+               ;; (let ((start (max 0 (- vlf-file-size vlf-batch-size))))
+               ;;   (while (and (< (- end start) n) (< n end))
+               ;;     (erase-buffer)
+               ;;     (vlf-tune-insert-file-contents-literally start end)
+               ;;     (goto-char (point-max))
+               ;;     (while (re-search-backward "[\n\C-m]" nil t)
+               ;;       (setq n (1- n)))
+               ;;     (vlf-tune-batch '(:raw))
+               ;;     (setq end start
+               ;;           start (max 0 (- end vlf-batch-size)))
+               ;;     (progress-reporter-update reporter
+               ;;                               (- vlf-file-size end))))
+               (when (< n end)
+                 (vlf-tune-batch '(:insert :encode))
+                 (vlf-move-to-chunk (- end vlf-batch-size) end)
+                 (goto-char (point-max))
+                 (setq success (vlf-re-search "[\n\C-m]" n t
+                                              reporter time))))))
+        (if font-lock (font-lock-mode 1))
+        (unless success
+          (vlf-with-undo-disabled
+           (vlf-move-to-chunk start-pos end-pos))
+          (goto-char pos)
+          (setq vlf-batch-size batch-size)
+          (message "Unable to find line"))
+        (run-hook-with-args 'vlf-after-batch-functions 'goto-line)))))
+
+(defun vlf-goto-line-hexl (n)
+  "Go to line N.  If N is negative, count from the end of file.
+Assume `hexl-mode' is active."
+  (vlf-tune-load '(:hexl :raw))
+  (if (< n 0)
+      (let ((hidden-bytes (+ vlf-file-size (* n hexl-bits))))
+        (setq hidden-bytes (- hidden-bytes (mod hidden-bytes
+                                                vlf-batch-size)))
+        (vlf-move-to-batch hidden-bytes)
+        (goto-char (point-max))
+        (forward-line (+ (round (- vlf-file-size
+                                   (min vlf-file-size
+                                        (+ hidden-bytes
+                                           vlf-batch-size)))
+                                hexl-bits)
+                         n)))
+    (let ((hidden-bytes (1- (* n hexl-bits))))
+      (setq hidden-bytes (- hidden-bytes (mod hidden-bytes
+                                              vlf-batch-size)))
+      (vlf-move-to-batch hidden-bytes)
+      (goto-char (point-min))
+      (forward-line (- n 1 (/ hidden-bytes hexl-bits))))))
+
+(defun vlf-query-replace (regexp to-string &optional delimited backward)
+  "Query replace over whole file matching REGEXP with TO-STRING.
+Third arg DELIMITED (prefix arg if interactive), if non-nil, replace
+only matches surrounded by word boundaries.  A negative prefix arg means
+replace BACKWARD."
+  (interactive (let ((common (query-replace-read-args
+                              (concat "Query replace over whole file"
+                                      (if current-prefix-arg
+                                          (if (eq current-prefix-arg '-)
+                                              " backward"
+                                            " word")
+                                        "")
+                                      " regexp")
+                              t)))
+                 (list (nth 0 common) (nth 1 common) (nth 2 common)
+                       (nth 3 common))))
+  (let ((not-automatic t))
+    (while (vlf-re-search regexp 1 backward)
+      (cond (not-automatic
+             (query-replace-regexp regexp to-string delimited
+                                   nil nil backward)
+             (if (eq 'automatic (lookup-key query-replace-map
+                                            (vector last-input-event)))
+                 (setq not-automatic nil)))
+            (backward (while (re-search-backward regexp nil t)
+                        (replace-match to-string)))
+            (t (while (re-search-forward regexp nil t)
+                 (replace-match to-string))))
+      (if (buffer-modified-p)
+          (save-buffer)))))
 
 (provide 'vlf-search)
 

vlf-setup.el

@@ -1,6 +1,6 @@
-;;; vlf-integrate.el --- VLF integration with other packages  -*- lexical-binding: t -*-
+;;; vlf-setup.el --- VLF integration with other packages  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2014 Free Software Foundation, Inc.
+;; Copyright (C) 2014-2015 Free Software Foundation, Inc.
 
 ;; Keywords: large files, integration
 ;; Author: Andrey Kotlarski <m00naticus@gmail.com>
@@ -50,6 +50,9 @@ Possible values are: nil to never use it;
   "Major modes which VLF will not be automatically applied to."
   :group 'vlf :type '(list symbol))
 
+(defvar dired-mode-map)
+(declare-function dired-get-file-for-visit "dired")
+
 (unless (fboundp 'file-size-human-readable)
   (defun file-size-human-readable (file-size)
     "Print FILE-SIZE in MB."
@@ -152,6 +155,6 @@ defined in FILE."
 (eval-after-load "dired"
   '(define-key dired-mode-map "V" 'dired-vlf))
 
-(provide 'vlf-integrate)
+(provide 'vlf-setup)
 
-;;; vlf-integrate.el ends here
+;;; vlf-setup.el ends here

vlf-tune.el

@@ -56,15 +56,20 @@ but don't change batch size.  If t, measure and change."
             (* 1000 (string-to-number (substring free match-from
                                                  (match-end 0))))))))
 
-(defcustom vlf-tune-max (let ((ram-size (vlf-tune-ram-size)))
-                          (if ram-size
-                              (/ ram-size 20)
-                            large-file-warning-threshold))
-  "Maximum batch size in bytes when auto tuning."
+(defcustom vlf-tune-max (max (let ((ram-size (vlf-tune-ram-size)))
+                               (if ram-size
+                                   (/ ram-size 20)
+                                 0))
+                             (if large-file-warning-threshold
+                                 large-file-warning-threshold
+                               0))
+  "Maximum batch size in bytes when auto tuning.
+Avoid increasing this after opening file with VLF."
   :group 'vlf :type 'integer)
 
-(defcustom vlf-tune-step (/ vlf-tune-max 1000)
-  "Step used for tuning in bytes."
+(defcustom vlf-tune-step (/ vlf-tune-max 10000)
+  "Step used for tuning in bytes.
+Avoid decreasing this after opening file with VLF."
   :group 'vlf :type 'integer)
 
 (defcustom vlf-tune-load-time 1.0
@@ -88,18 +93,35 @@ but don't change batch size.  If t, measure and change."
 
 (defvar vlf-tune-write-bps nil
   "Vector of bytes per second write measurements.")
-(make-variable-buffer-local 'vlf-tune-write-bps)
-(put 'vlf-tune-write-bps 'permanent-local t)
 
 (defvar vlf-tune-hexl-bps nil
   "Vector of bytes per second hexlify measurements.")
-(make-variable-buffer-local 'vlf-tune-hexl-bps)
-(put 'vlf-tune-hexl-bps 'permanent-local t)
 
 (defvar vlf-tune-dehexlify-bps nil
   "Vector of bytes per second dehexlify measurements.")
-(make-variable-buffer-local 'vlf-tune-dehexlify-bps)
-(put 'vlf-tune-dehexlify-bps 'permanent-local t)
+
+(defvar vlf-start-pos)
+(defvar hexl-bits)
+(defvar hexl-max-address)
+(declare-function hexl-line-displen "hexl")
+(declare-function dehexlify-buffer "hexl")
+
+(defun vlf-tune-copy-profile (from-buffer &optional to-buffer)
+  "Copy specific profile vectors of FROM-BUFFER to TO-BUFFER.
+If TO-BUFFER is nil, copy to current buffer."
+  (let (insert-bps insert-raw-bps encode-bps)
+    (with-current-buffer from-buffer
+      (setq insert-bps vlf-tune-insert-bps
+            insert-raw-bps vlf-tune-insert-raw-bps
+            encode-bps vlf-tune-encode-bps))
+    (if to-buffer
+        (with-current-buffer to-buffer
+          (setq vlf-tune-insert-bps insert-bps
+                vlf-tune-insert-raw-bps insert-raw-bps
+                vlf-tune-encode-bps encode-bps))
+      (setq vlf-tune-insert-bps insert-bps
+            vlf-tune-insert-raw-bps insert-raw-bps
+            vlf-tune-encode-bps encode-bps))))
 
 (defun vlf-tune-closest-index (size)
   "Get closest measurement index corresponding to SIZE."
@@ -111,9 +133,7 @@ but don't change batch size.  If t, measure and change."
 
 (defun vlf-tune-initialize-measurement ()
   "Initialize measurement vector."
-  (make-local-variable 'vlf-tune-max)
-  (make-local-variable 'vlf-tune-step)
-  (make-vector (/ vlf-tune-max vlf-tune-step) nil))
+  (make-vector (1- (/ vlf-tune-max vlf-tune-step)) nil))
 
 (defmacro vlf-tune-add-measurement (vec size time)
   "Add at an appropriate position in VEC new SIZE TIME measurement.
@@ -147,10 +167,11 @@ VEC is a vector of (mean time . count) elements ordered by size."
                               (- end start) (car result))
     (cdr result)))
 
-(defun vlf-tune-insert-file-contents-literally (start end)
-  "Insert raw file bytes START to END and save time it takes."
+(defun vlf-tune-insert-file-contents-literally (start end &optional file)
+  "Insert raw file bytes START to END and save time it takes.
+FILE if given is filename to be used, otherwise `buffer-file-name'."
   (let ((result (vlf-time (insert-file-contents-literally
-                           buffer-file-name nil start end))))
+                           (or file buffer-file-name) nil start end))))
     (vlf-tune-add-measurement vlf-tune-insert-raw-bps
                               (- end start) (car result))
     (cdr result)))
@@ -164,27 +185,44 @@ VEC is a vector of (mean time . count) elements ordered by size."
                               (cdr result) (car result))
     (cdr result)))
 
-(defun vlf-tune-write (start end append visit size)
+(defun vlf-tune-write (start end append visit size &optional file-name)
   "Save buffer and save time it takes.
 START, END, APPEND, VISIT have same meaning as in `write-region'.
-SIZE is number of bytes that are saved."
-  (let ((time (car (vlf-time (write-region start end buffer-file-name
-                                           append visit)))))
-    (vlf-tune-add-measurement vlf-tune-write-bps size time)))
+SIZE is number of bytes that are saved.
+FILE-NAME if given is to be used instead of `buffer-file-name'."
+  (let* ((file (or file-name buffer-file-name))
+         (time (car (vlf-time (write-region start end file append
+                                            visit)))))
+    (or (file-remote-p file) ;writing to remote files can include network copying
+        (vlf-tune-add-measurement vlf-tune-write-bps size time))))
+
+(defun vlf-hexl-adjust-addresses ()
+  "Adjust hexl address indicators according to `vlf-start-pos'."
+  (let ((pos (point))
+        (address vlf-start-pos))
+    (goto-char (point-min))
+    (while (re-search-forward "^[[:xdigit:]]+" nil t)
+      (replace-match (format "%08x" address))
+      (setq address (+ address hexl-bits)))
+    (goto-char pos)))
 
 (defun vlf-tune-hexlify ()
   "Activate `hexl-mode' and save time it takes."
-  (or (derived-mode-p 'hexl-mode)
-      (let ((time (car (vlf-time (hexl-mode)))))
+  (let* ((no-adjust (zerop vlf-start-pos))
+         (time (car (vlf-time (hexlify-buffer)
+                              (or no-adjust
+                                  (vlf-hexl-adjust-addresses))))))
+    (setq hexl-max-address (+ (* (/ (1- (buffer-size))
+                                    (hexl-line-displen)) 16) 15))
+    (or no-adjust
         (vlf-tune-add-measurement vlf-tune-hexl-bps
                                   hexl-max-address time))))
 
 (defun vlf-tune-dehexlify ()
   "Exit `hexl-mode' and save time it takes."
-  (if (derived-mode-p 'hexl-mode)
-      (let ((time (car (vlf-time (hexl-mode-exit)))))
-        (vlf-tune-add-measurement vlf-tune-dehexlify-bps
-                                  hexl-max-address time))))
+  (let ((time (car (vlf-time (dehexlify-buffer)))))
+    (vlf-tune-add-measurement vlf-tune-dehexlify-bps
+                              hexl-max-address time)))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;; tuning
@@ -233,7 +271,8 @@ unless DONT-APPROXIMATE is t."
                      `(aset ,vec ,index
                             (vlf-tune-approximate-nearby ,vec ,index))
                    `(vlf-tune-approximate-nearby ,vec ,index)))
-               (t val)))))
+               (t val)))
+     most-positive-fixnum))
 
 (defmacro vlf-tune-get-vector (key)
   "Get vlf-tune vector corresponding to KEY."
@@ -268,7 +307,7 @@ If it is number, stop as soon as cumulative time gets equal or above."
         (let ((bps (if (consp el)
                        (vlf-tune-assess (car el) (cadr el) index
                                         approximate)
-                     (vlf-tune-assess el 1 index approximate))))
+                     (vlf-tune-assess el 1.0 index approximate))))
           (if (zerop bps)
               (throw 'result nil)
             (setq time (+ time (/ size bps)))
@@ -332,55 +371,57 @@ MIN and MAX specify interval of indexes to search."
           (setq vlf-batch-size (* (1+ left-idx) vlf-tune-step)))))))
 
 (defun vlf-tune-linear (types max-idx)
-  "Adjust `vlf-batch-size' to optimal value using linear search, \
-optimizing over TYPES up to MAX-IDX."
+  "Adjust `vlf-batch-size' to optimal known value using linear search.
+Optimize over TYPES up to MAX-IDX."
   (let ((best-idx 0)
         (best-bps 0)
-        (idx 0)
-        (none-missing t))
-    (while (and none-missing (< idx max-idx))
-      (let ((bps (vlf-tune-score types idx)))
-        (cond ((null bps)
-               (setq vlf-batch-size (* (1+ idx) vlf-tune-step)
-                     none-missing nil))
-              ((< best-bps bps) (setq best-idx idx
-                                      best-bps bps))))
+        (idx 0))
+    (while (< idx max-idx)
+      (let ((bps (vlf-tune-score types idx t)))
+        (and bps (< best-bps bps)
+             (setq best-idx idx
+                   best-bps bps)))
       (setq idx (1+ idx)))
-    (or (not none-missing)
-        (setq vlf-batch-size (* (1+ best-idx) vlf-tune-step)))))
+    (setq vlf-batch-size (* (1+ best-idx) vlf-tune-step))))
 
-(defun vlf-tune-batch (types &optional linear)
+(defun vlf-tune-batch (types &optional linear file)
   "Adjust `vlf-batch-size' to optimal value optimizing on TYPES.
 TYPES is alist of elements that may be of form (type coef) or
-non list values in which case coeficient is assumed 1.
+non list values in which case coefficient is assumed 1.
 Types can be :insert, :raw, :encode, :write, :hexl or :dehexlify.
 If LINEAR is non nil, use brute-force.  In case requested measurement
 is missing, stop search and set `vlf-batch-size' to this value.
+FILE if given is filename to be used, otherwise `buffer-file-name'.
 Suitable for multiple batch operations."
   (if (eq vlf-tune-enabled t)
       (let ((max-idx (1- (/ (min vlf-tune-max
                                  (/ (1+ vlf-file-size) 2))
                             vlf-tune-step))))
-        (cond (linear (vlf-tune-linear types max-idx))
-              ((file-remote-p buffer-file-name)
-               (vlf-tune-conservative types))
-              ((<= 1 max-idx)
-               (if (< max-idx 3)
-                   (vlf-tune-conservative types (/ max-idx 2))
-                 (vlf-tune-binary types 0 max-idx)))))))
+        (if linear
+            (vlf-tune-linear types max-idx)
+          (let ((batch-size vlf-batch-size))
+            (cond ((file-remote-p (or file buffer-file-name))
+                   (vlf-tune-conservative types))
+                  ((<= 1 max-idx)
+                   (if (< max-idx 3)
+                       (vlf-tune-conservative types (/ max-idx 2))
+                     (vlf-tune-binary types 0 max-idx))))
+            (if (= batch-size vlf-batch-size) ;local maxima?
+                (vlf-tune-linear types max-idx)))))))
 
 (defun vlf-tune-optimal-load (types &optional min-idx max-idx)
   "Get best batch size according to existing measurements over TYPES.
 Best considered where primitive operations total is closest to
 `vlf-tune-load-time'.  If MIN-IDX and MAX-IDX are given,
 confine search to this region."
-  (if vlf-tune-enabled
+  (if (eq vlf-tune-enabled t)
       (progn
-        (setq max-idx (min (or max-idx vlf-tune-max)
+        (setq min-idx (max 0 (or min-idx 0))
+              max-idx (min (or max-idx vlf-tune-max)
                            (1- (/ (min vlf-tune-max
                                        (/ (1+ vlf-file-size) 2))
                                   vlf-tune-step))))
-        (let* ((idx (max 0 (or min-idx 0)))
+        (let* ((idx min-idx)
                (best-idx idx)
                (best-time-diff vlf-tune-load-time)
                (all-less t)

vlf-write.el

@@ -29,6 +29,13 @@
 
 (require 'vlf-base)
 
+(defcustom vlf-save-in-place 'ask
+  "Should VLF save in place when additional adjustment of file content\
+is needed."
+  :group 'vlf :type '(choice (const :tag "Always when applicable" t)
+                             (const :tag "Ask when applicable" 'ask)
+                             (const :tag "Never" nil)))
+
 (defun vlf-write ()
   "Write current chunk to file.  Always return true to disable save.
 If changing size of chunk, shift remaining file content."
@@ -48,45 +55,59 @@ If changing size of chunk, shift remaining file content."
           (progn (vlf-tune-write nil nil vlf-start-pos t
                                  (vlf-tune-encode-length (point-min)
                                                          (point-max)))
+                 (if hexl (vlf-tune-hexlify))
                  (setq vlf-file-size (vlf-get-file-size
                                       buffer-file-truename)
-                       vlf-end-pos vlf-file-size)
-                 (vlf-update-buffer-name))
+                       vlf-end-pos vlf-file-size))
         (let* ((region-length (vlf-tune-encode-length (point-min)
                                                       (point-max)))
                (size-change (- vlf-end-pos vlf-start-pos
                                region-length)))
           (if (zerop size-change)
-              (vlf-tune-write nil nil vlf-start-pos t
-                              (- vlf-end-pos vlf-start-pos))
-            (let ((tramp-verbose (if (boundp 'tramp-verbose)
-                                     (min tramp-verbose 2)))
-                  (pos (point))
-                  (font-lock font-lock-mode))
+              (progn (vlf-tune-write nil nil vlf-start-pos t
+                                     (- vlf-end-pos vlf-start-pos))
+                     (if hexl (vlf-tune-hexlify)))
+            (let ((pos (point))
+                  (font-lock font-lock-mode)
+                  (batch-size vlf-batch-size)
+                  time)
               (font-lock-mode 0)
-              (let ((batch-size vlf-batch-size)
-                    (time (float-time)))
+              (if (or (file-remote-p buffer-file-name)
+                      (if (eq vlf-save-in-place 'ask)
+                          (y-or-n-p "File content needs be adjusted\
+ till end.  Use temporary copy of the whole file (slower but safer)? ")
+                        (not vlf-save-in-place)))
+                  (let ((file-tmp (make-temp-file "vlf")))
+                    (setq time (float-time))
+                    (copy-file buffer-file-name file-tmp t t t t)
+                    (if (< 0 size-change)
+                        (vlf-file-shift-back size-change region-length
+                                             file-tmp)
+                      (vlf-file-shift-forward (- size-change)
+                                              region-length file-tmp))
+                    (rename-file file-tmp buffer-file-name t))
+                (setq time (float-time))
                 (if (< 0 size-change)
                     (vlf-file-shift-back size-change region-length)
-                  (vlf-file-shift-forward (- size-change) region-length))
-                (if font-lock (font-lock-mode 1))
-                (setq vlf-batch-size batch-size)
-                (vlf-move-to-chunk-2 vlf-start-pos
-                                     (if (< (- vlf-end-pos vlf-start-pos)
-                                            vlf-batch-size)
-                                         (+ vlf-start-pos vlf-batch-size)
-                                       vlf-end-pos))
-                (vlf-update-buffer-name)
-                (goto-char pos)
-                (message "Save took %f seconds" (- (float-time) time)))))))
-      (if hexl (vlf-tune-hexlify)))
+                  (vlf-file-shift-forward (- size-change)
+                                          region-length)))
+              (if font-lock (font-lock-mode 1))
+              (setq vlf-batch-size batch-size)
+              (vlf-move-to-chunk-2 vlf-start-pos
+                                   (if (< (- vlf-end-pos vlf-start-pos)
+                                          vlf-batch-size)
+                                       (+ vlf-start-pos vlf-batch-size)
+                                     vlf-end-pos))
+              (goto-char pos)
+              (message "Save took %f seconds" (- (float-time) time)))))))
     (run-hook-with-args 'vlf-after-batch-functions 'write))
   t)
 
-(defun vlf-file-shift-back (size-change write-size)
+(defun vlf-file-shift-back (size-change write-size &optional file)
   "Shift file contents SIZE-CHANGE bytes back.
-WRITE-SIZE is byte length of saved chunk."
-  (vlf-tune-write nil nil vlf-start-pos t write-size)
+WRITE-SIZE is byte length of saved chunk.
+FILE if given is filename to be used, otherwise `buffer-file-name'."
+  (vlf-tune-write nil nil vlf-start-pos (if file nil t) write-size file)
   (let ((read-start-pos vlf-end-pos)
         (coding-system-for-write 'no-conversion)
         (reporter (make-progress-reporter "Adjusting file content..."
@@ -94,33 +115,36 @@ WRITE-SIZE is byte length of saved chunk."
                                           vlf-file-size)))
     (vlf-with-undo-disabled
      (while (vlf-shift-batch read-start-pos (- read-start-pos
-                                               size-change))
+                                               size-change)
+                             file)
        (setq read-start-pos (+ read-start-pos vlf-batch-size))
        (progress-reporter-update reporter read-start-pos))
      ;; pad end with space
      (erase-buffer)
-     (vlf-verify-size t)
+     (vlf-verify-size t file)
      (insert-char 32 size-change))
     (vlf-tune-write nil nil (- vlf-file-size size-change)
-                    t size-change)
+                    (if file nil t) size-change file)
     (progress-reporter-done reporter)))
 
-(defun vlf-shift-batch (read-pos write-pos)
+(defun vlf-shift-batch (read-pos write-pos file)
   "Read `vlf-batch-size' bytes from READ-POS and write them \
-back at WRITE-POS.  Return nil if EOF is reached, t otherwise."
+back at WRITE-POS using FILE.
+Return nil if EOF is reached, t otherwise."
   (erase-buffer)
-  (vlf-verify-size t)
-  (vlf-tune-batch '(:raw :write))
-  (let ((read-end (min (+ read-pos vlf-batch-size) vlf-file-size)))
-    (vlf-tune-insert-file-contents-literally read-pos read-end)
-    (vlf-tune-write nil nil write-pos 0 (- read-end read-pos))
+  (vlf-verify-size t file)
+  (vlf-tune-batch '(:raw :write) nil file) ;insert speed over temp write file may defer wildly
+  (let ((read-end (min (+ read-pos vlf-batch-size) vlf-file-size))) ;compared to the original file
+    (vlf-tune-insert-file-contents-literally read-pos read-end file)
+    (vlf-tune-write nil nil write-pos 0 (- read-end read-pos) file)
     (< read-end vlf-file-size)))
 
-(defun vlf-file-shift-forward (size-change write-size)
+(defun vlf-file-shift-forward (size-change write-size &optional file)
   "Shift file contents SIZE-CHANGE bytes forward.
 WRITE-SIZE is byte length of saved chunk.
+FILE if given is filename to be used, otherwise `buffer-file-name'.
 Done by saving content up front and then writing previous batch."
-  (vlf-tune-batch '(:raw :write))
+  (vlf-tune-batch '(:raw :write) nil file)
   (let ((read-size (max vlf-batch-size size-change))
         (read-pos vlf-end-pos)
         (write-pos vlf-start-pos)
@@ -129,8 +153,8 @@ Done by saving content up front and then writing previous batch."
                                           vlf-file-size)))
     (vlf-with-undo-disabled
      (when (vlf-shift-batches read-size read-pos write-pos
-                              write-size t)
-       (vlf-tune-batch '(:raw :write))
+                              write-size t file)
+       (vlf-tune-batch '(:raw :write) nil file)
        (setq write-pos (+ read-pos size-change)
              read-pos (+ read-pos read-size)
              write-size read-size
@@ -138,8 +162,8 @@ Done by saving content up front and then writing previous batch."
        (progress-reporter-update reporter write-pos)
        (let ((coding-system-for-write 'no-conversion))
          (while (vlf-shift-batches read-size read-pos write-pos
-                                   write-size nil)
-           (vlf-tune-batch '(:raw :write))
+                                   write-size nil file)
+           (vlf-tune-batch '(:raw :write) nil file)
            (setq write-pos (+ read-pos size-change)
                  read-pos (+ read-pos read-size)
                  write-size read-size
@@ -148,25 +172,27 @@ Done by saving content up front and then writing previous batch."
     (progress-reporter-done reporter)))
 
 (defun vlf-shift-batches (read-size read-pos write-pos write-size
-                                    hide-read)
+                                    hide-read file)
   "Append READ-SIZE bytes of file starting at READ-POS.
 Then write initial buffer content to file at WRITE-POS.
 WRITE-SIZE is byte length of saved chunk.
 If HIDE-READ is non nil, temporarily hide literal read content.
+FILE if given is filename to be used, otherwise `buffer-file-name'.
 Return nil if EOF is reached, t otherwise."
-  (vlf-verify-size t)
+  (vlf-verify-size t file)
   (let ((read-more (< read-pos vlf-file-size))
         (start-write-pos (point-min))
         (end-write-pos (point-max)))
     (when read-more
       (goto-char end-write-pos)
       (vlf-tune-insert-file-contents-literally
-       read-pos (min vlf-file-size (+ read-pos read-size))))
+       read-pos (min vlf-file-size (+ read-pos read-size)) file))
     ;; write
     (if hide-read ; hide literal region if user has to choose encoding
         (narrow-to-region start-write-pos end-write-pos))
     (vlf-tune-write start-write-pos end-write-pos write-pos
-                    (or (not read-more) 0) write-size)
+                    (or (and (not read-more) (not file)) 0)
+                    write-size file)
     (delete-region start-write-pos end-write-pos)
     (if hide-read (widen))
     read-more))

vlf.el

@@ -1,13 +1,13 @@
 ;;; vlf.el --- View Large Files  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2006, 2012-2014 Free Software Foundation, Inc.
+;; Copyright (C) 2006, 2012-2015 Free Software Foundation, Inc.
 
-;; Version: 1.6
+;; Version: 1.7
 ;; Keywords: large files, utilities
 ;; Maintainer: Andrey Kotlarski <m00naticus@gmail.com>
 ;; Authors: 2006 Mathias Dahl <mathias.dahl@gmail.com>
 ;;          2012 Sam Steingold <sds@gnu.org>
-;;          2013-2014 Andrey Kotlarski <m00naticus@gmail.com>
+;;          2013-2015 Andrey Kotlarski <m00naticus@gmail.com>
 ;; URL: https://github.com/m00natic/vlfi
 
 ;; This file is free software; you can redistribute it and/or modify
@@ -31,7 +31,7 @@
 ;; which provides several commands for moving around, searching,
 ;; comparing and editing selected part of file.
 ;; To have it offered when opening large files:
-;; (require 'vlf-integrate)
+;; (require 'vlf-setup)
 
 ;; This package was inspired by a snippet posted by Kevin Rodgers,
 ;; showing how to use `insert-file-contents' to extract part of a
@@ -53,12 +53,20 @@ One argument is supplied that specifies current action.  Possible
 values are: `write', `ediff', `occur', `search', `goto-line'."
   :group 'vlf :type 'hook)
 
+(defcustom vlf-batch-size-remote 1024
+  "Defines size (in bytes) of a batch of file data when accessed remotely."
+  :group 'vlf :type 'integer)
+
+(defvar hexl-bits)
+
 (autoload 'vlf-write "vlf-write" "Write current chunk to file." t)
 (autoload 'vlf-re-search-forward "vlf-search"
   "Search forward for REGEXP prefix COUNT number of times." t)
 (autoload 'vlf-re-search-backward "vlf-search"
   "Search backward for REGEXP prefix COUNT number of times." t)
 (autoload 'vlf-goto-line "vlf-search" "Go to line." t)
+(autoload 'vlf-query-replace "vlf-search"
+  "Query replace regexp over whole file." t)
 (autoload 'vlf-occur "vlf-occur"
   "Make whole file occur style index for REGEXP." t)
 (autoload 'vlf-toggle-follow "vlf-follow"
@@ -79,6 +87,7 @@ values are: `write', `ediff', `occur', `search', `goto-line'."
         (vlf-change-batch-size t)))
     (define-key map "s" 'vlf-re-search-forward)
     (define-key map "r" 'vlf-re-search-backward)
+    (define-key map "%" 'vlf-query-replace)
     (define-key map "o" 'vlf-occur)
     (define-key map "[" 'vlf-beginning-of-file)
     (define-key map "]" 'vlf-end-of-file)
@@ -98,7 +107,11 @@ values are: `write', `ediff', `occur', `search', `goto-line'."
 
 (define-minor-mode vlf-mode
   "Mode to browse large files in."
-  :lighter " VLF" :group 'vlf :keymap vlf-prefix-map
+  :group 'vlf :keymap vlf-prefix-map
+  :lighter (:eval (format " VLF[%d/%d](%s)"
+                          (ceiling vlf-end-pos vlf-batch-size)
+                          (ceiling vlf-file-size vlf-batch-size)
+                          (file-size-human-readable vlf-file-size)))
   (cond (vlf-mode
          (set (make-local-variable 'require-final-newline) nil)
          (add-hook 'write-file-functions 'vlf-write nil t)
@@ -125,14 +138,25 @@ values are: `write', `ediff', `occur', `search', `goto-line'."
          (remove-hook 'write-file-functions 'vlf-write t)
          (remove-hook 'after-change-major-mode-hook
                       'vlf-keep-alive t)
-         (let ((hexl (derived-mode-p 'hexl-mode)))
-           (if hexl (hexl-mode-exit))
-           (let ((pos (+ vlf-start-pos (position-bytes (point)))))
+         (if (derived-mode-p 'hexl-mode)
+             (let ((line (/ (1+ vlf-start-pos) hexl-bits))
+                   (pos (point)))
+               (if (consp buffer-undo-list)
+                   (setq buffer-undo-list nil))
+               (vlf-with-undo-disabled
+                (let ((inhibit-read-only t))
+                  (insert-file-contents-literally buffer-file-name
+                                                  t nil nil t)
+                  (hexlify-buffer)))
+               (set-buffer-modified-p nil)
+               (goto-char (point-min))
+               (forward-line line)
+               (forward-char pos))
+           (let ((pos (+ vlf-start-pos (position-bytes (point))))
+                 (inhibit-read-only t))
              (vlf-with-undo-disabled
               (insert-file-contents buffer-file-name t nil nil t))
-             (goto-char (byte-to-position pos)))
-           (if hexl (hexl-mode)))
-         (rename-buffer (file-name-nondirectory buffer-file-name) t))
+             (goto-char (byte-to-position pos)))))
         (t (setq vlf-mode t))))
 
 (defun vlf-keep-alive ()
@@ -152,8 +176,10 @@ Return newly created buffer."
     (set-buffer vlf-buffer)
     (set-visited-file-name file)
     (set-buffer-modified-p nil)
-    (if (or minimal (file-remote-p file))
-        (set (make-local-variable 'vlf-batch-size) 1024))
+    (cond (minimal
+           (set (make-local-variable 'vlf-batch-size) 1024))
+          ((file-remote-p file)
+           (set (make-local-variable 'vlf-batch-size) vlf-batch-size-remote)))
     (vlf-mode 1)
     (when minimal                 ;restore batch size to default value
       (kill-local-variable 'vlf-batch-size)
@@ -170,7 +196,7 @@ When prefix argument is negative
   (interactive "p")
   (vlf-verify-size)
   (vlf-tune-load (if (derived-mode-p 'hexl-mode)
-                     '(:hexl :dehexlify :insert :encode)
+                     '(:hexl :raw)
                    '(:insert :encode)))
   (let* ((end (min (+ vlf-end-pos (* vlf-batch-size (abs append)))
                    vlf-file-size))
@@ -189,7 +215,7 @@ When prefix argument is negative
   (if (zerop vlf-start-pos)
       (error "Already at BOF"))
   (vlf-tune-load (if (derived-mode-p 'hexl-mode)
-                     '(:hexl :dehexlify :insert :encode)
+                     '(:hexl :raw)
                    '(:insert :encode)))
   (let* ((start (max 0 (- vlf-start-pos (* vlf-batch-size (abs prepend)))))
          (end (if (< prepend 0)
@@ -241,6 +267,16 @@ When prefix argument is negative
        (if (and vlf-mode (pos-visible-in-window-p (point-min)))
            (progn (vlf-prev-batch 1)
                   (goto-char (point-max)))
+         ad-do-it))
+
+     (defadvice hexl-mode-exit (around vlf-hexl-mode-exit
+                                       activate compile)
+       "Exit `hexl-mode' gracefully in case `vlf-mode' is active."
+       (if (and vlf-mode (not (buffer-modified-p)))
+           (vlf-with-undo-disabled
+            (erase-buffer)
+            ad-do-it
+            (vlf-move-to-chunk-2 vlf-start-pos vlf-end-pos))
          ad-do-it))))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -260,7 +296,7 @@ with the prefix argument DECREASE it is halved."
    (list (read-number "Size in bytes: "
                       (vlf-tune-optimal-load
                        (if (derived-mode-p 'hexl-mode)
-                           '(:hexl :dehexlify :insert :encode)
+                           '(:hexl :raw)
                          '(:insert :encode))))))
   (setq vlf-batch-size size)
   (vlf-move-to-batch vlf-start-pos))
@@ -269,7 +305,7 @@ with the prefix argument DECREASE it is halved."
   "Jump to beginning of file content."
   (interactive)
   (vlf-tune-load (if (derived-mode-p 'hexl-mode)
-                     '(:hexl :dehexlify :insert :encode)
+                     '(:hexl :raw)
                    '(:insert :encode)))
   (vlf-move-to-batch 0))
 
@@ -278,7 +314,7 @@ with the prefix argument DECREASE it is halved."
   (interactive)
   (vlf-verify-size)
   (vlf-tune-load (if (derived-mode-p 'hexl-mode)
-                     '(:hexl :dehexlify :insert :encode)
+                     '(:hexl :raw)
                    '(:insert :encode)))
   (vlf-move-to-batch vlf-file-size))
 
@@ -296,7 +332,7 @@ Ask for confirmation if NOCONFIRM is nil."
   "Go to to chunk N."
   (interactive "nGoto to chunk: ")
   (vlf-tune-load (if (derived-mode-p 'hexl-mode)
-                     '(:hexl :dehexlify :insert :encode)
+                     '(:hexl :raw)
                    '(:insert :encode)))
   (vlf-move-to-batch (* (1- n) vlf-batch-size)))
 
@@ -306,16 +342,15 @@ Ask for confirmation if NOCONFIRM is nil."
       (error "Save or discard your changes first")
     t))
 
-(defun vlf-move-to-batch (start &optional minimal)
+(defun vlf-move-to-batch (start)
   "Move to batch determined by START.
-Adjust according to file start/end and show `vlf-batch-size' bytes.
-When given MINIMAL flag, skip non important operations."
+Adjust according to file start/end and show `vlf-batch-size' bytes."
   (vlf-verify-size)
   (let* ((start (max 0 start))
          (end (min (+ start vlf-batch-size) vlf-file-size)))
     (if (= vlf-file-size end)          ; re-adjust start
         (setq start (max 0 (- end vlf-batch-size))))
-    (vlf-move-to-chunk start end minimal)))
+    (vlf-move-to-chunk start end)))
 
 (defun vlf-next-batch-from-point ()
   "Display batch of file data starting from current point."