Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 390 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.6059. Min Range: -3.6059

Max Act: 3.6059. Min Act: -0.0000

Data Index: 1512422 (The Pile)

Max Activating Token Index: 657

Text #1

Max Range: 3.6059. Min Range: -3.6059

Max Act: 3.3632. Min Act: -0.0000

Data Index: 1882863 (The Pile)

Max Activating Token Index: 206

Text #2

Max Range: 3.6059. Min Range: -3.6059

Max Act: 3.1749. Min Act: -0.0000

Data Index: 139533 (The Pile)

Max Activating Token Index: 299

Text #3

Max Range: 3.6059. Min Range: -3.6059

Max Act: 3.1596. Min Act: -0.0000

Data Index: 1591697 (The Pile)

Max Activating Token Index: 138

Text #4

Max Range: 3.6059. Min Range: -3.6059

Max Act: 3.4164. Min Act: -0.0000

Data Index: 212790 (The Pile)

Max Activating Token Index: 958

Text #5

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.9295. Min Act: -0.0000

Data Index: 284269 (The Pile)

Max Activating Token Index: 364

Text #6

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.9468. Min Act: -0.0000

Data Index: 1151593 (The Pile)

Max Activating Token Index: 811

Text #7

Max Range: 3.6059. Min Range: -3.6059

Max Act: 3.0434. Min Act: -0.0000

Data Index: 1981993 (The Pile)

Max Activating Token Index: 867

Text #8

Max Range: 3.6059. Min Range: -3.6059

Max Act: 3.0212. Min Act: -0.0000

Data Index: 1965561 (The Pile)

Max Activating Token Index: 1014

Text #9

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.7563. Min Act: -0.0000

Data Index: 1850654 (The Pile)

Max Activating Token Index: 769

Text #10

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.8272. Min Act: -0.0000

Data Index: 1854121 (The Pile)

Max Activating Token Index: 855

Text #11

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.7620. Min Act: -0.0000

Data Index: 1325721 (The Pile)

Max Activating Token Index: 228

Text #12

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.6239. Min Act: -0.0000

Data Index: 1010112 (The Pile)

Max Activating Token Index: 782

Text #13

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.7465. Min Act: -0.0000

Data Index: 192724 (The Pile)

Max Activating Token Index: 288

Text #14

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.6966. Min Act: -0.0000

Data Index: 37364 (The Pile)

Max Activating Token Index: 174

Text #15

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.8002. Min Act: -0.0000

Data Index: 257365 (The Pile)

Max Activating Token Index: 103

Text #16

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.7833. Min Act: -0.0000

Data Index: 1802585 (The Pile)

Max Activating Token Index: 887

Text #17

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.3958. Min Act: -0.0000

Data Index: 1599418 (The Pile)

Max Activating Token Index: 839

Text #18

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.8721. Min Act: -0.0000

Data Index: 1182735 (The Pile)

Max Activating Token Index: 348

Text #19

Max Range: 3.6059. Min Range: -3.6059

Max Act: 2.6195. Min Act: -0.0000

Data Index: 1280368 (The Pile)

Max Activating Token Index: 121