Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4605 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 8.3163. Min Range: -8.3163

Max Act: 8.3163. Min Act: -0.0000

Data Index: 139243 (The Pile)

Max Activating Token Index: 18

Text #1

Max Range: 8.3163. Min Range: -8.3163

Max Act: 7.2190. Min Act: -0.0000

Data Index: 236925 (The Pile)

Max Activating Token Index: 172

Text #2

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.9531. Min Act: -0.0000

Data Index: 1357282 (The Pile)

Max Activating Token Index: 214

Text #3

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.6690. Min Act: -0.0000

Data Index: 553130 (The Pile)

Max Activating Token Index: 383

Text #4

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.8681. Min Act: -0.0000

Data Index: 1436610 (The Pile)

Max Activating Token Index: 271

Text #5

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.7234. Min Act: -0.0000

Data Index: 1731625 (The Pile)

Max Activating Token Index: 414

Text #6

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.6755. Min Act: -0.0000

Data Index: 390037 (The Pile)

Max Activating Token Index: 22

Text #7

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.6753. Min Act: -0.0000

Data Index: 626208 (The Pile)

Max Activating Token Index: 279

Text #8

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.4545. Min Act: -0.0000

Data Index: 816406 (The Pile)

Max Activating Token Index: 496

Text #9

Max Range: 8.3163. Min Range: -8.3163

Max Act: 5.9470. Min Act: -0.0000

Data Index: 445553 (The Pile)

Max Activating Token Index: 794

Text #10

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.8409. Min Act: -0.0000

Data Index: 1152220 (The Pile)

Max Activating Token Index: 84

Text #11

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.8059. Min Act: -0.0000

Data Index: 506100 (The Pile)

Max Activating Token Index: 343

Text #12

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.4344. Min Act: -0.0000

Data Index: 1136581 (The Pile)

Max Activating Token Index: 111

Text #13

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.6005. Min Act: -0.0000

Data Index: 1191930 (The Pile)

Max Activating Token Index: 343

Text #14

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.3191. Min Act: -0.0000

Data Index: 457634 (The Pile)

Max Activating Token Index: 151

Text #15

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.1288. Min Act: -0.0000

Data Index: 359772 (The Pile)

Max Activating Token Index: 217

Text #16

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.4242. Min Act: -0.0000

Data Index: 338050 (The Pile)

Max Activating Token Index: 208

Text #17

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.2231. Min Act: -0.0000

Data Index: 107466 (The Pile)

Max Activating Token Index: 270

Text #18

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.1942. Min Act: -0.0000

Data Index: 1389218 (The Pile)

Max Activating Token Index: 410

Text #19

Max Range: 8.3163. Min Range: -8.3163

Max Act: 6.1801. Min Act: -0.0000

Data Index: 786205 (The Pile)

Max Activating Token Index: 317