Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1558 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.6400. Min Act: -0.0000

Data Index: 839415 (C4 (Web Text))

Max Activating Token Index: 610

Text #1

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.3807. Min Act: -0.0000

Data Index: 549347 (C4 (Web Text))

Max Activating Token Index: 471

Text #2

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.4084. Min Act: -0.0000

Data Index: 564577 (C4 (Web Text))

Max Activating Token Index: 572

Text #3

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.4112. Min Act: -0.0000

Data Index: 709542 (C4 (Web Text))

Max Activating Token Index: 96

Text #4

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.4287. Min Act: -0.0000

Data Index: 1182177 (C4 (Web Text))

Max Activating Token Index: 493

Text #5

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.3415. Min Act: -0.0000

Data Index: 610503 (C4 (Web Text))

Max Activating Token Index: 143

Text #6

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.3276. Min Act: -0.0000

Data Index: 843694 (C4 (Web Text))

Max Activating Token Index: 448

Text #7

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.2527. Min Act: -0.0000

Data Index: 1240570 (C4 (Web Text))

Max Activating Token Index: 690

Text #8

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.2604. Min Act: -0.0000

Data Index: 1216068 (C4 (Web Text))

Max Activating Token Index: 314

Text #9

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.1673. Min Act: -0.0000

Data Index: 1149180 (C4 (Web Text))

Max Activating Token Index: 1018

Text #10

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.1932. Min Act: -0.0000

Data Index: 319000 (C4 (Web Text))

Max Activating Token Index: 124

Text #11

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.1002. Min Act: -0.0000

Data Index: 1287791 (C4 (Web Text))

Max Activating Token Index: 948

Text #12

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.1314. Min Act: -0.0000

Data Index: 1246072 (C4 (Web Text))

Max Activating Token Index: 663

Text #13

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.1327. Min Act: -0.0000

Data Index: 344098 (C4 (Web Text))

Max Activating Token Index: 737

Text #14

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.0735. Min Act: -0.0000

Data Index: 558400 (C4 (Web Text))

Max Activating Token Index: 976

Text #15

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.0496. Min Act: -0.0000

Data Index: 1083042 (C4 (Web Text))

Max Activating Token Index: 128

Text #16

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.0659. Min Act: -0.0000

Data Index: 91800 (C4 (Web Text))

Max Activating Token Index: 607

Text #17

Max Range: 1.6400. Min Range: -1.6400

Max Act: 0.9928. Min Act: -0.0000

Data Index: 767454 (C4 (Web Text))

Max Activating Token Index: 826

Text #18

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.0691. Min Act: -0.0000

Data Index: 1224711 (C4 (Web Text))

Max Activating Token Index: 84

Text #19

Max Range: 1.6400. Min Range: -1.6400

Max Act: 1.0337. Min Act: -0.0000

Data Index: 308103 (C4 (Web Text))

Max Activating Token Index: 270