Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5060 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.4333. Min Range: -5.4333

Max Act: 5.4333. Min Act: -0.0000

Data Index: 494662 (The Pile)

Max Activating Token Index: 810

Text #1

Max Range: 5.4333. Min Range: -5.4333

Max Act: 5.2226. Min Act: -0.0000

Data Index: 952978 (The Pile)

Max Activating Token Index: 825

Text #2

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.9362. Min Act: -0.0000

Data Index: 490440 (The Pile)

Max Activating Token Index: 949

Text #3

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.6949. Min Act: -0.0000

Data Index: 1444366 (The Pile)

Max Activating Token Index: 527

Text #4

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.8314. Min Act: -0.0000

Data Index: 1930199 (The Pile)

Max Activating Token Index: 926

Text #5

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.6956. Min Act: -0.0000

Data Index: 153697 (The Pile)

Max Activating Token Index: 342

Text #6

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.3264. Min Act: -0.0000

Data Index: 1354919 (The Pile)

Max Activating Token Index: 735

Text #7

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.0276. Min Act: -0.0000

Data Index: 1318580 (The Pile)

Max Activating Token Index: 970

Text #8

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.2925. Min Act: -0.0000

Data Index: 1155867 (The Pile)

Max Activating Token Index: 275

Text #9

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.2699. Min Act: -0.0000

Data Index: 886285 (The Pile)

Max Activating Token Index: 226

Text #10

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.2432. Min Act: -0.0000

Data Index: 599572 (The Pile)

Max Activating Token Index: 366

Text #11

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.1310. Min Act: -0.0000

Data Index: 388275 (The Pile)

Max Activating Token Index: 682

Text #12

Max Range: 5.4333. Min Range: -5.4333

Max Act: 3.8926. Min Act: -0.0000

Data Index: 993922 (The Pile)

Max Activating Token Index: 218

Text #13

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.1456. Min Act: -0.0000

Data Index: 1284863 (The Pile)

Max Activating Token Index: 715

Text #14

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.1260. Min Act: -0.0000

Data Index: 787464 (The Pile)

Max Activating Token Index: 740

Text #15

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.1480. Min Act: -0.0000

Data Index: 730433 (The Pile)

Max Activating Token Index: 289

Text #16

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.0996. Min Act: -0.0000

Data Index: 351239 (The Pile)

Max Activating Token Index: 856

Text #17

Max Range: 5.4333. Min Range: -5.4333

Max Act: 3.9804. Min Act: -0.0000

Data Index: 303026 (The Pile)

Max Activating Token Index: 296

Text #18

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.2794. Min Act: -0.0000

Data Index: 1709610 (The Pile)

Max Activating Token Index: 104

Text #19

Max Range: 5.4333. Min Range: -5.4333

Max Act: 4.2774. Min Act: -0.0000

Data Index: 918255 (The Pile)

Max Activating Token Index: 1010