Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2668 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5625. Min Act: -0.0004

Data Index: 1115716 (C4 (Web Text))

Max Activating Token Index: 345

Text #1

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4820. Min Act: -0.0004

Data Index: 953973 (C4 (Web Text))

Max Activating Token Index: 142

Text #2

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5060. Min Act: -0.0004

Data Index: 671602 (C4 (Web Text))

Max Activating Token Index: 985

Text #3

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5240. Min Act: -0.0004

Data Index: 1326851 (C4 (Web Text))

Max Activating Token Index: 372

Text #4

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4989. Min Act: -0.0004

Data Index: 675866 (C4 (Web Text))

Max Activating Token Index: 158

Text #5

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5494. Min Act: -0.0004

Data Index: 825127 (C4 (Web Text))

Max Activating Token Index: 261

Text #6

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4918. Min Act: -0.0004

Data Index: 563908 (C4 (Web Text))

Max Activating Token Index: 575

Text #7

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4592. Min Act: -0.0004

Data Index: 506368 (C4 (Web Text))

Max Activating Token Index: 444

Text #8

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5054. Min Act: -0.0004

Data Index: 556993 (C4 (Web Text))

Max Activating Token Index: 537

Text #9

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5377. Min Act: -0.0004

Data Index: 14341 (C4 (Web Text))

Max Activating Token Index: 489

Text #10

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4843. Min Act: -0.0004

Data Index: 1199980 (C4 (Web Text))

Max Activating Token Index: 366

Text #11

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5112. Min Act: -0.0004

Data Index: 734227 (C4 (Web Text))

Max Activating Token Index: 686

Text #12

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5263. Min Act: -0.0004

Data Index: 78089 (C4 (Web Text))

Max Activating Token Index: 497

Text #13

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5270. Min Act: -0.0004

Data Index: 1094660 (C4 (Web Text))

Max Activating Token Index: 260

Text #14

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4897. Min Act: -0.0004

Data Index: 628771 (C4 (Web Text))

Max Activating Token Index: 705

Text #15

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.5084. Min Act: -0.0004

Data Index: 442688 (C4 (Web Text))

Max Activating Token Index: 462

Text #16

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4738. Min Act: -0.0004

Data Index: 1286766 (C4 (Web Text))

Max Activating Token Index: 173

Text #17

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4897. Min Act: -0.0004

Data Index: 69379 (C4 (Web Text))

Max Activating Token Index: 510

Text #18

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4446. Min Act: -0.0004

Data Index: 1236843 (C4 (Web Text))

Max Activating Token Index: 913

Text #19

Max Range: 0.5625. Min Range: -0.5625

Max Act: 0.4417. Min Act: -0.0004

Data Index: 69704 (C4 (Web Text))

Max Activating Token Index: 617