Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2764 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.2452. Min Act: -0.0002

Data Index: 316479 (C4 (Web Text))

Max Activating Token Index: 752

Text #1

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1939. Min Act: -0.0002

Data Index: 1353847 (C4 (Web Text))

Max Activating Token Index: 154

Text #2

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1862. Min Act: -0.0001

Data Index: 577192 (C4 (Web Text))

Max Activating Token Index: 21

Text #3

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1743. Min Act: -0.0002

Data Index: 1047130 (C4 (Web Text))

Max Activating Token Index: 286

Text #4

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1912. Min Act: -0.0001

Data Index: 1142737 (C4 (Web Text))

Max Activating Token Index: 490

Text #5

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1632. Min Act: -0.0001

Data Index: 985819 (C4 (Web Text))

Max Activating Token Index: 879

Text #6

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1501. Min Act: -0.0001

Data Index: 718803 (C4 (Web Text))

Max Activating Token Index: 843

Text #7

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1528. Min Act: -0.0001

Data Index: 460479 (C4 (Web Text))

Max Activating Token Index: 310

Text #8

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1397. Min Act: -0.0001

Data Index: 604920 (C4 (Web Text))

Max Activating Token Index: 623

Text #9

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1494. Min Act: -0.0002

Data Index: 1346634 (C4 (Web Text))

Max Activating Token Index: 964

Text #10

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1472. Min Act: -0.0001

Data Index: 287863 (C4 (Web Text))

Max Activating Token Index: 562

Text #11

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1466. Min Act: -0.0001

Data Index: 946126 (C4 (Web Text))

Max Activating Token Index: 609

Text #12

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1351. Min Act: -0.0002

Data Index: 412205 (C4 (Web Text))

Max Activating Token Index: 699

Text #13

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1521. Min Act: -0.0002

Data Index: 413444 (C4 (Web Text))

Max Activating Token Index: 729

Text #14

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1422. Min Act: -0.0001

Data Index: 75489 (C4 (Web Text))

Max Activating Token Index: 793

Text #15

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1431. Min Act: -0.0002

Data Index: 648654 (C4 (Web Text))

Max Activating Token Index: 535

Text #16

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1462. Min Act: -0.0002

Data Index: 503963 (C4 (Web Text))

Max Activating Token Index: 142

Text #17

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1433. Min Act: -0.0002

Data Index: 1084954 (C4 (Web Text))

Max Activating Token Index: 912

Text #18

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1386. Min Act: -0.0001

Data Index: 108911 (C4 (Web Text))

Max Activating Token Index: 114

Text #19

Max Range: 0.2452. Min Range: -0.2452

Max Act: 0.1365. Min Act: -0.0001

Data Index: 589540 (C4 (Web Text))

Max Activating Token Index: 862