Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4200 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.9320. Min Range: -1.9320

Max Act: 1.9320. Min Act: -0.0000

Data Index: 159774 (The Pile)

Max Activating Token Index: 529

Text #1

Max Range: 1.9320. Min Range: -1.9320

Max Act: 1.3761. Min Act: -0.0000

Data Index: 1041823 (The Pile)

Max Activating Token Index: 632

Text #2

Max Range: 1.9320. Min Range: -1.9320

Max Act: 1.1405. Min Act: -0.0000

Data Index: 1527101 (The Pile)

Max Activating Token Index: 23

Text #3

Max Range: 1.9320. Min Range: -1.9320

Max Act: 1.0548. Min Act: -0.0000

Data Index: 411560 (The Pile)

Max Activating Token Index: 748

Text #4

Max Range: 1.9320. Min Range: -1.9320

Max Act: 1.0158. Min Act: -0.0000

Data Index: 215788 (The Pile)

Max Activating Token Index: 512

Text #5

Max Range: 1.9320. Min Range: -1.9320

Max Act: 1.0722. Min Act: -0.0000

Data Index: 194279 (The Pile)

Max Activating Token Index: 1001

Text #6

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.9871. Min Act: -0.0000

Data Index: 1016312 (The Pile)

Max Activating Token Index: 884

Text #7

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.9084. Min Act: -0.0000

Data Index: 312024 (The Pile)

Max Activating Token Index: 245

Text #8

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8959. Min Act: -0.0000

Data Index: 889739 (The Pile)

Max Activating Token Index: 774

Text #9

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.9112. Min Act: -0.0000

Data Index: 1767743 (The Pile)

Max Activating Token Index: 267

Text #10

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8885. Min Act: -0.0000

Data Index: 1557798 (The Pile)

Max Activating Token Index: 846

Text #11

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8929. Min Act: -0.0000

Data Index: 1365815 (The Pile)

Max Activating Token Index: 473

Text #12

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8780. Min Act: -0.0000

Data Index: 42377 (The Pile)

Max Activating Token Index: 795

Text #13

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8632. Min Act: -0.0000

Data Index: 1633502 (The Pile)

Max Activating Token Index: 601

Text #14

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8365. Min Act: -0.0000

Data Index: 1791412 (The Pile)

Max Activating Token Index: 575

Text #15

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8671. Min Act: -0.0000

Data Index: 1724830 (The Pile)

Max Activating Token Index: 120

Text #16

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8387. Min Act: -0.0000

Data Index: 1770805 (The Pile)

Max Activating Token Index: 102

Text #17

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8412. Min Act: -0.0000

Data Index: 266839 (The Pile)

Max Activating Token Index: 966

Text #18

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.8274. Min Act: -0.0000

Data Index: 295994 (The Pile)

Max Activating Token Index: 335

Text #19

Max Range: 1.9320. Min Range: -1.9320

Max Act: 0.7939. Min Act: -0.0000

Data Index: 401885 (The Pile)

Max Activating Token Index: 381