Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4184 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.4693. Min Act: -0.0000

Data Index: 684672 (C4 (Web Text))

Max Activating Token Index: 825

Text #1

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.1583. Min Act: -0.0000

Data Index: 11877 (C4 (Web Text))

Max Activating Token Index: 661

Text #2

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.1514. Min Act: -0.0000

Data Index: 79677 (C4 (Web Text))

Max Activating Token Index: 461

Text #3

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.1290. Min Act: -0.0000

Data Index: 70437 (C4 (Web Text))

Max Activating Token Index: 830

Text #4

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0524. Min Act: -0.0000

Data Index: 1124531 (C4 (Web Text))

Max Activating Token Index: 101

Text #5

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0272. Min Act: -0.0000

Data Index: 482407 (C4 (Web Text))

Max Activating Token Index: 905

Text #6

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0221. Min Act: -0.0000

Data Index: 649925 (C4 (Web Text))

Max Activating Token Index: 523

Text #7

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0211. Min Act: -0.0000

Data Index: 211761 (C4 (Web Text))

Max Activating Token Index: 493

Text #8

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0494. Min Act: -0.0000

Data Index: 1140624 (C4 (Web Text))

Max Activating Token Index: 10

Text #9

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0630. Min Act: -0.0000

Data Index: 562834 (C4 (Web Text))

Max Activating Token Index: 472

Text #10

Max Range: 1.4693. Min Range: -1.4693

Max Act: 0.9041. Min Act: -0.0000

Data Index: 688485 (C4 (Web Text))

Max Activating Token Index: 211

Text #11

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0499. Min Act: -0.0000

Data Index: 669694 (C4 (Web Text))

Max Activating Token Index: 7

Text #12

Max Range: 1.4693. Min Range: -1.4693

Max Act: 0.9831. Min Act: -0.0000

Data Index: 758155 (C4 (Web Text))

Max Activating Token Index: 582

Text #13

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0178. Min Act: -0.0000

Data Index: 1008473 (C4 (Web Text))

Max Activating Token Index: 106

Text #14

Max Range: 1.4693. Min Range: -1.4693

Max Act: 1.0025. Min Act: -0.0000

Data Index: 265579 (C4 (Web Text))

Max Activating Token Index: 913

Text #15

Max Range: 1.4693. Min Range: -1.4693

Max Act: 0.9883. Min Act: -0.0000

Data Index: 132559 (C4 (Web Text))

Max Activating Token Index: 625

Text #16

Max Range: 1.4693. Min Range: -1.4693

Max Act: 0.9771. Min Act: -0.0000

Data Index: 595886 (C4 (Web Text))

Max Activating Token Index: 85

Text #17

Max Range: 1.4693. Min Range: -1.4693

Max Act: 0.9091. Min Act: -0.0000

Data Index: 289361 (C4 (Web Text))

Max Activating Token Index: 205

Text #18

Max Range: 1.4693. Min Range: -1.4693

Max Act: 0.9933. Min Act: -0.0000

Data Index: 462353 (C4 (Web Text))

Max Activating Token Index: 310

Text #19

Max Range: 1.4693. Min Range: -1.4693

Max Act: 0.9573. Min Act: -0.0000

Data Index: 1283433 (C4 (Web Text))

Max Activating Token Index: 448